Maximum number of syxdump positions?

[jjonas]

Hi,

I trying to pick parameter values from a sysex dump that's over 1000 bytes long (a DSI Tetra combo edit buffer dump), is that too much for syxdump_pos? It's not possible to request a smaller slice. Requesting and receiving 439 bytes (normal mode edit buffer dump) seems to work ok, but with the <1000 dump I'm getting strange results at least from byte 137 onwards for some reason. Parameters values that should be in bytes one after the other are not; it appears as if some bytes are missing in between.

For the moment I'd just like to determine whether or not it's a syxdump_pos limitation that might be causing the behaviour.

[TK.]

Hi,

syxdump pos goes up to 12bit (=4096 bytes)

But you've to be careful with CPU load - than more events are listening to SysEx streams, than higher the chance to get input buffer overflows, which would lead to some missing bytes (as you noticed).

Best Regards, Thorsten.

[jjonas]

I uncommented all RECEIVERs and SENDERs except for one RECEIVER that receives the dump. My setup is such that the dump from the synth goes through a SEQv4 (v4+.095) before reaching the NG. If I bypass the SEQv4+ and put the dump directly into the NG, there doesn't seem to be dropped bytes, but if the dump goes through SEQv4+, some bytes (somewhere after the 1000 bytes mark) would seem to drop. The router on the relevant ports on the SEQv4+ is all channels in, all channels out (not sure if that affects sysex).

Is it expected that routing the dump through the SEQv4+ could cause problems? The seq's info page shows CPU load 15% during the dump and no registered drops.

UPDATE: I noticed today (13.2.) that even though I get the dump directly into the NG (and with only one receiver event) some bytes are jumping around between requests. So it might not have anything to do with SEQv4. I tried sending a 4000 byte test file from the computer to NG, that seemed to work ok, maybe I'll have to try different midi cables...

UPDATE2: I tried 4 different midi cables, no difference. The next thing to try is to dump the data on the computer and diff the files to see if it's the synth that's sending inconsistent dumps.

Edited February 13, 2020 by jjonas

[m.str]

I might be too late to join this thread, but I am currently running a setup, that seems similar to your project regarding large SysEx Dumps and multiple RECEIVERS/SENDERS listening to a received dump.

Right now I am quite confused on how your up-to-4000-Bytes-long dump can actually, under certain circumstances, be received correctly. The dumps I request from my Hardware Synths are "only" 90-850 Bytes long and there is ALWAYS a certain point of cut-off, whereafter all remaining bytes get lost and the listening EVENTs are being initialized with a value of 0.

And of course I always traced back my SysEx-related problems to two facts:

a) the number of Events listening to syxdump_postitions is too high (the Encoders of my hardware MIDI controller need to be initialized with syxdump_pos values, a whole set of encoders consists of 64 Encoders minimum, switched on/off with the banking mechanism so a total of 32 Encoders should be active at once) -> Each Encoder requires two Events listening to the same syxdump_pos though..

And yes, this results in what TK has mentioned, I am having repeated notifications on timeout problems, always on the MIDI IN ports where the incoming dumps sent by the hardware synths come in.

b) On the other hand I was always assured that SysEx messages that huge could NEVER be processed in total, because of the Input Buffer Limitations and the resulting bottleneck issues.

Before reading this I was actually about to ask if there is a nice way to priorise the Reception of a Sysex Stream until the 0xf7 has arrived, meaning that even if the maximum buffer size has been filled there could somehow be a workaround with priorising the Reading of these large SysEx files (at least until some other break statement has been reached..)

Maybe something in the likes of this: https://github.com/midibox/mios32/tree/master/apps/tutorials/025_sysex_and_eeprom

But since you could progress with your giant dumps I came to the conclusion, that the main problems really are the listening events and they seem to causes severer problems.. Is that correct? How did you get along with your project, did you find a way to re-incorporate the listening events?

Greetings,

Micha

[jjonas]

I didn't make any progress after the last message I wrote, unfortunately.

I don't remember the details anymore, but based on what I wrote above — that sending 4000 bytes from the computer works ok, but sending a somewhat smaller number of bytes from the synth doesn't — I think I concluded that it's too much work for too unlikely gains.

One of the reasons for this lack of faith is that the synth in question, DSI Tetra, has some problems, or at least some unpolished characteristics, with the combo mode. That's why I think it's possible that the syxdump problems might be because of the implementation of the synth, and that there's nothing I can do on the MBNG side. After all, a big test file dump from the computer to the MBNG works ok.

[m.str]

Thanks for sharing your experiences on this.

I think it's kind of comforting to know that the dump reception from a synth might be problematic in general, though it obviously does not solve both of our problems :)

Another unfortunate realisation came to me after I had reduced the syxdump_pos-listening EVENTs by half. It did not have any apparent effect on the Input Buffer Overflow, the stream was cut at the same position as before.