Advanced task management

[Sauraen]

Hi TK,

Right now I'm in a course on real-time operating systems, and I'd like to lay the foundation for a more stable and higher-performance MIDIbox Quad Genesis, so I want to start doing some fancy things. This is a simplified version of the picture, but if I can figure this part out I should be able to do the whole thing.

I have a task that checks whether certain buffers need to be filled, and if so, reads into them from a file on the SD card. This task needs to run every so often at medium priority, and there are other low-priority tasks running.

I see that currently the SPI transfer from the SD card is accomplished via DMA, but the CPU still busy-waits during the transfer. Instead, once the DMA has started, I want the task to be suspended; and then once the DMA Complete interrupt happens, I want the task to resume from the next line of code (go back to the FATFS driver and process the data, etc.).

Two questions:

1. Looking at FreeRTOS, I can call vTaskSuspend after the DMA has started, and then from the DMA Complete ISR call xTaskResumeFromISR. The problem with this is that it simply marks the task as ready, it doesn't actually context-switch back to it. So for instance, if a low-priority task was running, this would still make the SD card task wait for up to a whole ms before a tick interrupt gave it control again. I want the OS to reevaluate the priorities after the ISR--in effect, force a virtual "tick" right after returning from the DMA Complete ISR. Do you know if this is possible? Maybe somehow overriding the tick timer to expire very soon (but it'd have to get put back to its normal length immediately by the OS)?
2. To implement any of this, I'd have to at least modify mios32_spi.c, and possibly other MIOS32 components as well. I can put my changes in #ifdef SAURAEN_DMA_TEST or something like this so they're guaranteed not to mess up everyone else's use of MIOS32 even if they're buggy--but I'm not sure if you want me touching those files anyway. Is there any better way?

Thanks,

Sauraen

[lis0r]

http://www.freertos.org/FreeRTOS_Support_Forum_Archive/June_2007/freertos_portYIELD_FROM_ISR_1752080.html explains how you need to approach this.

I think you should use the xTaskNotify() family of functions, instead - the kernel has already implemented blocking, there's no point reinventing the wheel.

 

[Sauraen]

3 hours ago, lis0r said:

http://www.freertos.org/FreeRTOS_Support_Forum_Archive/June_2007/freertos_portYIELD_FROM_ISR_1752080.html explains how you need to approach this.

I think you should use the xTaskNotify() family of functions, instead - the kernel has already implemented blocking, there's no point reinventing the wheel.

Thanks for the tip! After a bit of research it looks like vPortYieldFromISR() is what I was looking for. Evidently, this sets up a "pending interrupt" that's ignored when in kernel mode (including in the ISR) but happens as soon as the CPU goes back to user mode (starting to execute the low-priority task again). Then the kernel will task-switch to the suspended medium-priority task. I hope it works!

Also, is there any reason that this is not implemented in the existing MIOS32? That all MIDIboxes busy-wait during SD card and DIN/DOUT SPI transfers?

[TK.]

The reason is, that all MIOS32 sources (under mios32/) work independent from FreeRTOS; I don't want to introduce special FreeRTOS mechanisms there to simplify ports to other operating systems in future.

Another reason is, that in my use cases I haven't seen the requirement to allow the execution of same or lower prio tasks if the tasks which loads/stores from/to SD Card is stalled.

If you want to try this in MIOS32, please don't use "SAURAEN_DMA_TEST" as a switch, and don't call FreeRTOS functions directly.
(if it's only a test, you wouldn't submit the changes to the repository anyhow...)

instead use a generic names, similar to the way how I made the integration of Mutexes optional (search for MUTEX)

Best Regards, Thorsten. 

P.S.: DIN/DOUT SPI transfers use the DMA callback hook - they are not waiting!

[Sauraen]

Hi TK,

I understand about mios32/ being platform-independent, and I'd prefer not to modify critical mios32 components anyway. As far as committing, all my changes do have to eventually go up, so that people who buy the MBHP_Genesis board can get the software.

So you're saying have the SPI code call generic-looking functions that I create (e.g. MIOS32_TaskSuspend or something) and then define these in some file that's included in the FreeRTOS folder? What and where should I define those functions for other platforms? I'll take a look at your implementation of Mutexes, maybe that'll help.

[TK.]

No, I wouldn't provide a MIOS32_TaskSuspend function, but integrate an optional (#define based) call into the MIOS32_SPI module which would initiate the suspend for SPI transfers instead - similar to the way how I integrated Mutexes.

You are asking specific questions where it would be easier for me to implement a generic change based on a prototype that you provide for your specific use case. Means: try to modify the current code until you are satisfied, explain your changes (don't commit them to the repository), and then we've a good basis for further discussions. This makes it much easier for both sides.

Best Regards, Thorsten.

[lis0r]

I still think it would make more sense to utilise an existing blocking synchronisation primitive than try to hack around the scheduler, especially if the intent is to be OS agnostic.

Personally, I'd use queued I/O. You'd have a shared request queue, and a receive queue per task. The process would be like follows:

 

In the requesting task:

1) Build a transfer descriptor. This describes the data to read, where to write it, and specifies a receive queue to be notified.

2) Queue the descriptor on the request queue.

3) If the queue isn't running, in a critical section configure and start the request on head of the queue.

4) Perform a blocking read on the receive queue.

 

In the DMA ISR:

1) Pop the head of the request queue, and write a notice of completion to the receive queue it describes. This should cause the task to wake up.

2) Check if there is another request.

3) If so, initiate it and return.

4) If not, mark the queue as not running and return.

 

This gives you efficient blocking I/O without the overhead of another task. It's also worth checking if someone has already done the work of wrapping up queued I/O for you - it's certainly a feature in my pet RTOS.

[TK.]

This would be some kind of gateway to the SD Card; I agree that this would be the right choice for this use case, but the existing FILE->FatFS->MIOS32_SDCARD APIs couldn't be used for this purpose.

Instead, a dedicated SD Card handler would be required which knows from which sectors the data can be read (or written).

Actually very similar to the SD Card Sampler project: http://svnmios.midibox.org/filedetails.php?repname=svn.mios32&path=%2Ftrunk%2Fapps%2Fsynthesizers%2FSD+card+sample+player%2Fapp.c

which caches the "file positions" and can read them via MIOS32_SDCARD_SectorRead

The queue handling would require an alternative SDCARD_SectorRead function (implemented at application level) which uses the DMA callback mechanism as you described.

The good thing: this dedicated handling doesn't require any change in existing drivers.
And it's definitely the best choice performance-wise - if we ignore the fact that it requires some more code at application level (but this code could be "hidden" in a driver ;-)

Best Regards, Thorsten.

[Sauraen]

On 3/14/2016 at 7:09 PM, lis0r said:

Personally, I'd use queued I/O.

Let me say a few words about the application. There will be a number of buffers containing data which is being streamed from the SD card. A mid-priority task comes around every so often and checks if any queues need to be filled (they're double-buffered). If so, I want it to start the DMA transfer and then let the CPU do other things while that completes, and then continue. I would think this kind of thing is a pretty common use case.

I'd be happy to write a new SD card driver that uses the FreeRTOS wait -> notify mechanism I'm talking about (not least because it would avoid introducing new bugs into MIOS32). But I do need the functionality of the FILE and FatFS drivers, and the files I'm streaming from may be arbitrarily large (bigger than a sector), so I can't just get a pointer to the first sector and read from there.

The reason I was talking about adding this to the generic MIOS32 SD card functions is that other than the portability issue, I would think that most applications would want the behavior of the CPU being freed while a SD card DMA transfer was in progress. Of course there might be some applications where you want it to block during the transfer, but that would seem to be the exception.

As you pointed out TK, DIN/DOUT gets around this by using callbacks, and implementing basically the entire functionality of the DIO system in those callbacks. I can't do this because I need the data to trickle back up through FatFS and FILE to my routine. I have little to no idea what FatFS is doing with the data it gets back from the SD card, but I'm under the impression it's nontrivial. (My application can't guarantee that an entire transfer would be from one sector, for instance--it might overlap two.)

I think TK's idea of implementing something like the Mutexes is the best so far, but I'll have to take a closer look before I can get anywhere with it.

[TK.]

Quote

The reason I was talking about adding this to the generic MIOS32 SD card functions is that other than the portability issue, I would think that most applications would want the behavior of the CPU being freed while a SD card DMA transfer was in progress.

I assume that only a minority of applications need this.

During SD Card transfers FreeRTOS will switch to other higher prio tasks automatically within 1 mS (preemptive multitasking is enabled). If you organize the task priorities accordingly, there isn't a problem with the blocking behaviour. E.g. I don't see a need for MBSEQ, MBNG, MBCV

Best Regards, Thorsten.

[lis0r]

Read the appropriate FAT entries at file open, translate from file offset to cluster numbers when filling out the descriptor, and voila: queued, streaming, asynchronous file I/O, without desktop-levels of spurious tasks. Think embedded! You shouldn't have *any* "mid-priority task"s that "comes around every so often and checks" *anything* - polling is ugly and inefficient.

[Sauraen]

16 hours ago, TK. said:

I assume that only a minority of applications need this.

During SD Card transfers FreeRTOS will switch to other higher prio tasks automatically within 1 mS (preemptive multitasking is enabled). If you organize the task priorities accordingly, there isn't a problem with the blocking behaviour. E.g. I don't see a need for MBSEQ, MBNG, MBCV

Best Regards, Thorsten.

I know you know your applications better than I do (obviously), so I'll take your word for it. But in my application, it would be likely that a new block would have to be transferred from the SD card every 10 ms or so, so blocking for up to 1 ms (and then losing another up to 1 ms when the task switches back) is wasting an average of 10% of the CPU time.

16 hours ago, lis0r said:

Read the appropriate FAT entries at file open, translate from file offset to cluster numbers when filling out the descriptor, and voila

I really don't know anything about low-level filesystem stuff, and I don't feel like I should have to rewrite the entire driver just to avoid adding 3 lines of FreeRTOS code to MIOS32. I'm not going to make any changes to the public MIOS32 source unless TK approves them, but there's got to be a better solution than a whole new driver that's application-specific.

16 hours ago, lis0r said:

Think embedded! You shouldn't have *any* "mid-priority task"s that "comes around every so often and checks" *anything* - polling is ugly and inefficient.

The reason this is a "mid-priority task" is because there's another task, with a new priority level I defined called "insane", which has to run every few microseconds to push new DAC data to the YM2612s (there's four of them). This could take half the CPU time itself, depending on how much the user is playing. This function (it's not even a task, just an ISR) is what uses up the buffer data and flags buffers as needing to be filled. Whether those flags are in a small number of places to check or on a queue somewhere is besides the point. Clearly, some task with lower priority than this, but higher priority than things like DIN/DOUT, has to check these things and start transfers from the SD card. Problem is, if the user is trying to play too much, there will be no time other than during the SD card DMA transfers when the CPU can be free to do all other tasks!

What you described above with mutexes is fine as a solution for waking up the SD card thread immediately after the audio thread signals there's new data to be read. But the question here is how to put the SD card thread itself to sleep when the DMA transfer has started and wake it up again when it is complete. I have the answer--the FreeRTOS functions discussed earlier--the question is how to integrate them into MIOS32 without breaking its platform independence.

[lis0r]

Having to push data out every few microseconds sounds realtime. You should probably be doing that in a timer interrupt. The non-realtime part of the code should be converting the data from the on-disk format to a representation that makes real time processing easier. That's assuming you can't figure out a way of getting a DMA engine to do the job for you. From the description of your current scheme, it will be exceptionally fragile.

At the very least, you still shouldn't be polling - you should be able to work out when you'll need more data. Schedule a timer to initiate the reads, don't waste CPU time polling.

[TK.]

before we drift into an academical discussion: Sauraean, I added the hooks into the MIOS32_SDCARD driver for you, so that you can experiment at application level w/o touching MIOS32 code.

The change is available in the SVN repository, please update.

MIOS32_SDCARD_SectorRead (and SectorWrite) now contain:

  // read 512 bytes via DMA
#ifdef MIOS32_SDCARD_TASK_SUSPEND_HOOK
  MIOS32_SPI_TransferBlock(MIOS32_SDCARD_SPI, NULL, buffer, 512, MIOS32_SDCARD_TASK_RESUME_HOOK);
  MIOS32_SDCARD_TASK_SUSPEND_HOOK();
#else
  MIOS32_SPI_TransferBlock(MIOS32_SDCARD_SPI, NULL, buffer, 512, NULL);
#endif

so, once a function for MIOS32_SDCARD_TASK_SUSPEND_HOOK has been assigned, it will be called after MIOS32_SPI_TransferBlock. In addition, MIOS32_SDCARD_TASK_RESUME_HOOK will be called by the DMA callback

In your mios32_config.h file, add following code:

extern void APP_SDCARD_TaskSuspend(void);
extern void APP_SDCARD_TaskResume(void);
#define MIOS32_SDCARD_TASK_SUSPEND_HOOK APP_SDCARD_TaskSuspend
#define MIOS32_SDCARD_TASK_RESUME_HOOK  APP_SDCARD_TaskResume

And somewhere in your app.c file:

void APP_SDCARD_TaskSuspend(void)
{
  // suspend task here
}

void APP_SDCARD_TaskResume(void)
{
  // resume task here (will be called from DMA callback, it's an ISR)
}

Best Regards, Thorsten.

[Sauraen]

9 hours ago, TK. said:

before we drift into an academical discussion: Sauraean, I added the hooks into the MIOS32_SDCARD driver for you, so that you can experiment at application level w/o touching MIOS32 code.

Thanks TK! This is the kind of solution I was looking for. I will give it a try. It might be a couple weeks before I get to it though.