8 replies to this topic

[Duggle]

I'm having hard fault crashes in a routine that formally worked o.k
I suspect the problem may be insufficient stack.
In the task creation I specify configMinimalStack+1000 as the stack size parameter.
If I go any bigger than this I get "could not allocate block" etc (several times) on the Mios Studio console followed by an application crash.
Is there a way to make the stack bigger?

I set up a call to vTaskList() which prints out "high water mark" free stack space for each running task plus some other useful diagnostics, (used to work, and handy, too) but this crashes also, (maybe related?)

More generally, any advice for debugging when "hard faults" occur?

thanks

[TK.]

A very interesting topic for me, since I'm also searching for better diagnosis solutions to analyse the stack usage and especially hard faults.
Currently only the address at which the error happened will be output (this sometimes only works on a LCD, the message sometimes doesn't reach the MIOS Terminal - this could be solved in future by changing the stack pointer to a safe location, and to run only the important MIOS32 functions anymore (handling MIDI + special terminal commands)

Back to topic: each thread has it's own stack, which is configured with xTaskCreate.
Example:

  xTaskCreate(TASK_MIDI,              (signed portCHAR *)"MIDI",         configMINIMAL_STACK_SIZE, NULL, PRIORITY_TASK_MIDI, NULL);
  xTaskCreate(TASK_Period1mS,         (signed portCHAR *)"Period1mS",    configMINIMAL_STACK_SIZE, NULL, PRIORITY_TASK_PERIOD1MS, NULL);
  xTaskCreate(TASK_Period1mS_LowPrio, (signed portCHAR *)"Period1mS_LP", configMINIMAL_STACK_SIZE, NULL, PRIORITY_TASK_PERIOD1MS_LOW_PRIO, NULL);
  xTaskCreate(TASK_Pattern,           (signed portCHAR *)"Pattern",      configMINIMAL_STACK_SIZE, NULL, PRIORITY_TASK_PATTERN, &xPatternHandle);

here the stack size of each thread is the default value (configMINIMAL_STACK_SIZE) which is configured with MIOS32_MINIMAL_STACK_SIZE)

I also wrote some more informations about this parameter:

// Stack size for FreeRTOS tasks as defined by the programming model
// Note that each task maintains it's own stack!
// If you want to define a different stack size for your application tasks
// (-> xTaskCreate() function), keep in mind that it has to be divided by 4,
// since the stack width of ARM is 32bit.
// The FreeRTOS define "configMINIMAL_STACK_SIZE" is (MIOS32_MINIMAL_STACK_SIZE/4)
// it can be used in applications as well, e.g.
// xTaskCreate(TASK_Period1mS, (signed portCHAR *)"Period1mS", configMINIMAL_STACK_SIZE, NULL, PRIORITY_TASK_PERIOD1MS, NULL);
#define MIOS32_MINIMAL_STACK_SIZE 1100

This also means, that you could run a thread will a higher stack size, and all other threads with the default stack size to reduce the memory consumption:

  // run MIDI task with stack size 1500
  xTaskCreate(TASK_MIDI,              (signed portCHAR *)"MIDI",  1500/4, NULL, PRIORITY_TASK_MIDI, NULL);

Another possibility would be to increase the heap.
It's 10k (10*1024) by default, to use 14k instead write:

// reserved memory for FreeRTOS pvPortMalloc function
#define MIOS32_HEAP_SIZE 14*1024

into your mios32_config.h file.

Could you please show me your vTaskList() setup? It would be interesting for me as well - an adapted handler for MIOS32 could be added to modules/freertos_utils/freertos_utils.c, currently it only outputs the runtime stats.

In order to find out, how many bytes a thread consumes, just increase the stack size of the particular thread, increase the heap, determine the stack consumption, optimized the configuration.

Some more hints (we are almost out of topic now, but I guess that this is interesting to know):

With following lines you can determine the stack address in the current function:

  {
    int x;
    MIOS32_MIDI_SendDebugMessage("Stack: 0x%08x\n", (unsigned)&x);
  }

and will following code you can display the stack (e.g. to display the watermark bytes):

  {
    u8 *stackAddress = (u8 *)0x20082460; // e.g. determined with umm_info(1, NULL)
    u32 stackSize = MIOS32_MINIMAL_STACK_SIZE;
    MIOS32_MIDI_SendDebugHexDump(stackAddress, stackSize);
  }

Last but not least: I think that I should implement a less costly version of MIOS32_MIDI_SendDebugMessage - e.g. MIOS32_MIDI_SendDebugString - which doesn't consume so much stack by itself, and which allows to output a string with any length

Best Regards, Thorsten.

[TK.]

Btw: MIOS32_MIDI_SendDebugString is available now: http://www.midibox.o...c6d16d6ee501277

Bes Regards, Thorsten.

[Duggle]

The code below basically reformats the text so that the columns align nicely regardless if tabs are not supported.This is working for me again, now. I'm still getting failures (freezes) with other parts of the App, so I'll resort to the debugger with Eclipse.
I also received instructions on how to enable the Stateviewer kernel aware debugger plugin. It will be great if it works.
It has 2 windows that get updated upon break or pause:
Tasks: a more detailed data similar to vTaskList()
Queues: list details to do with the state of queues for which vQueueAddToRegistry() has been called.
Like I said: great if it works. I'll post back with results.

#define db MIOS32_MIDI_SendDebugMessage 
void Display_TaskList(){
int gap,x;
char *r,*s,*context1,*context2,*t,line[60];
 t=malloc(1000);
 vTaskList((signed char*)t);
 r=strtok_r((char*)t, "\n", &context1);
 while ®{
	// db®;
 	s=strtok_r((char*)r, " \t", &context2);
 	strcpy(line,s);						//name

 	gap=12-strlen(s);
 	for (x=0;x<gap;x++) strcat(line," ");  //pad to max 12 chars
 	s=strtok_r(NULL, " \t", &context2);
 	strcat(line,s);						//state

 	s=strtok_r(NULL, " \t", &context2);
 	gap=4-strlen(s);
 	for (x=0;x<gap;x++) strcat(line," ");  //right justify 12chars
 	strcat(line,s);						//priority

 	s=strtok_r(NULL, " \t", &context2);
 	gap=6-strlen(s);
 	for (x=0;x<gap;x++) strcat(line," ");  //right justify 12chars
 	strcat(line,s);						//free stack

 	s=strtok_r(NULL, " \t", &context2);
 	gap=4-strlen(s);
 	for (x=0;x<gap;x++) strcat(line," ");  //right justify 12chars
 	strcat(line,s);						//task id
 	db(line);

 	r=strtok_r(NULL,"\n",&context1);
 };
 free(t);
}

[Duggle]

Thanks for the tips, Thorsten.
By increasing the heap, I've been able to increase the stack in the (de)bugged task. It does not appear to be an out of stack situation.

Its a little frustrating the core becomes unresponsive after running a function I'm sure was working ok before... Anyhow I'll work through it.

Unfortunately the Eclipse OpenOCD setup does not work now that I am using Windows 7 64bit. It seems to be usb driver related. I have some options to use XP for debugging purposes.

[Duggle]

Well happily, Ive made progress with both debugger and debuggee!

I can confirm that Eclipse/OpenOCD 0.5.0 works fine with windows 7 x64. There is some very minor tweak to .cfg file using the interface and target files that come in the OpenOCD directories. If anyone is interested in the details of this esoterica please write to this thread or PM me.


Firstly the hard faults seem to have been caused by an array definition outside of the function that modifies it: by defining the array static has fixed the problem.
This is funny because I've not had this problem on other platforms.
The debugger stack display showed an execution address of 0xfffffffc followed by Hard fault handler followed by a series of hierarchical midi functions ending in a nonblocking midi tx function that was where it seemed to be stuck. Hence no midi error messages to MIOS Studio, and no LCD hard fault message.

Anyhow I thank debugging in Eclipse as a big help. I will persist in figuring out how to flash program through OpenOCD to improve the user experience even more.

[TK.]

Well happily, Ive made progress with both debugger and debuggee!

;-)

Firstly the hard faults seem to have been caused by an array definition outside of the function that modifies it: by defining the array static has fixed the problem.
This is funny because I've not had this problem on other platforms.

The difference between global and static (local) variables is, that the linker will locate local variables at the beginning of the SRAM together with some other, normally less critical, variables of other code modules -> see project_build/project.map

Example (array name: xxx)

 *(.bss .bss.* .gnu.linkonce.b.*)
 .bss.xxx       0x0000000010000000        0xc project_build/app.o
 .bss.uxMissedTicks
                0x000000001000000c        0x4 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.xNumOfOverflows
                0x0000000010000010        0x4 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.pxDelayedTaskList
                0x0000000010000014        0x4 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.xSchedulerRunning
                0x0000000010000018        0x4 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.uxTasksDeleted
                0x000000001000001c        0x4 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.xTasksWaitingTermination
                0x0000000010000020       0x14 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.xSuspendedTaskList
                0x0000000010000034       0x14 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o
 .bss.pxReadyTasksLists
                0x0000000010000048       0x64 project_build//Users/TK/svn/mios32/trunk/FreeRTOS/Source/tasks.o

If the array is declared as global variable, it will be located in a region which contains informations used by MIOS32 functions, such as:

                0x0000000010000518                xxx
 *fill*         0x0000000010000522        0x2 00
 COMMON         0x0000000010000524       0x40 project_build//Users/TK/svn/mios32/trunk/mios32/common/mios32_srio.o
                0x0000000010000524                mios32_srio_din_changed
                0x0000000010000534                mios32_srio_dout
                0x0000000010000544                mios32_srio_din_buffer
                0x0000000010000554                mios32_srio_din
 *fill*         0x0000000010000564        0x4 00
 COMMON         0x0000000010000568      0x300 project_build//Users/TK/svn/mios32/trunk/mios32/common/mios32_enc.o
                0x0000000010000568                enc_config
                0x0000000010000668                enc_state
 COMMON         0x0000000010000868       0x24 project_build//Users/TK/svn/mios32/trunk/mios32/common/mios32_lcd.o
                0x0000000010000868                font_bitmap
                0x0000000010000874                mios32_lcd_y
                0x0000000010000878                mios32_lcd_parameters
                0x0000000010000882                mios32_lcd_column
                0x0000000010000884                mios32_lcd_cursor_map
                0x0000000010000888                mios32_lcd_line
                0x000000001000088a                mios32_lcd_x

This could make the difference.

I think that I should isolate MIOS32 variables, and locate them to the beginning of SRAM to reduce the danger for overwritten critical variables on out-of-bounds array accesses.
This should make the upcoming diagnosis mode more stable.

My hope is that it will be even possible to enter bootloader mode from diagnosis mode, or to enter it automatically after 5 seconds if there is no user interaction anymore (e.g. triggered by a watchdog reset)

Best Regards, Thorsten.

[Duggle]

Some kind of diagnostic output, followed by a controlled reset (or vice versa) in the case of hard faults is a great idea!

With regard to my bug, the funny thing is, I didn't do anything wrong with array bounds, I haven't changed the code that writes to the array, it works.
It does make sense to what I was seeing that important system memory was getting clobbered: the manifestations where not always identical.

[Duggle]

Well, great news the (free) StateViewer plugin for Eclipse works!


In the picture I have only one queue (registered as "Console"). In the right window it shows that this queue is blocking "ConsoleT" as it waits for commands that are enqueued by the debug callback (comand line parser).
The "Min Free Stack" field shows only a maximum of 255 in bytes.
This is in contrast with the vTaskList() output which shows in min stack in 32bit words:


The instructions on how to download and install the free plugin is here:
http://www.highinteg...WER_Plug-in.pdf