Durisian Posted January 25, 2008 Report Posted January 25, 2008 So I'm experimenting with developing a PC editor for my pedal board project. At the moment i'm just reading a syx file downloaded from a bankstickI'm hoping someone can enlighten me as to the 7-bit scrambling.I have assumed it's the first byte of 8 that contains the first bit of the following 7.0ABCDEFG, 0AAAAAAA, 0BBBBBBB, 0CCCCCCC, 0DDDDDDDD, 0EEEEEEE, 0FFFFFFF, 0GGGGGGG.So to get the value of Bankstick address 0x0000Given that the header is 11 bytes long, it should be:value = ((12thByte & b01000000) << 1) | 13thByteCan anyon help me out?Cheers! Quote
TK. Posted January 25, 2008 Report Posted January 25, 2008 No, this kind of scrambling isn't used by MIOS, instead the bits are packed without de-positioning.You will find two useful perl scripts, which include the algorithms, in the MIOS source code package under tools/hex2syx.pl and tools/syx2asm.plBest Regards, Thorsten. Quote
Durisian Posted January 30, 2008 Author Report Posted January 30, 2008 whoa.... my head hurts...they really are scrambledAre there some notes that explain the concepts of the algorithms?I'm having some trouble following the math.Cheers! Quote
Durisian Posted January 31, 2008 Author Report Posted January 31, 2008 Alright, I've got the basic idea sorted. It's actually fairly straight forward.Nothing like some sleep to clear your head.Now where's my copy of Java for Dummies ;D Quote
TK. Posted January 31, 2008 Report Posted January 31, 2008 There is a Java based scrambling routine in the MIOS Studio package - but it doesn't descramble.Unfortunately Java is not like e, where you can easily define a bitvector with unlimited length, pack the 8bit value array into this vector, and unpack a 7bit value array from the vector. Thats basically the approach. In "e" a single line of code, in Java/C... hm... perhaps some more.Best Regards, Thorsten. Quote
Durisian Posted February 1, 2008 Author Report Posted February 1, 2008 haha, yeah I discovered that the hard way. :(my second attempt is to grab 1 bit at a time and place it directly in the correct byte, it's pretty ugly but works so far. Quote
TK. Posted February 1, 2008 Report Posted February 1, 2008 With Java (where memory consumption doesn't matter), you could maybe also do it the following way: convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.Pseudo code: string := "" foreach byte string := string + decimal_to_binary(byte) while lenth(string) && (length(string) >= 7) { SendMIDI(binary_to_decimal(string[0:6])) string := string[7:end] }(this is no official programming language, I only want to demonstrate the algorithm...)Best Regards, Thorsten. Quote
audiocommander Posted February 21, 2008 Report Posted February 21, 2008 Hello number-crunching friends!I'm also trying to figure that out for a patch editor written in C; however, I am not sure about the length of the returned sysEx header and would appreciate some clarification. I get these (hex-) values when I request the first bankstick's content:[tt]f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69[/tt]from the syx2asm.pl I see the header is:[tt]f0 00 00 7e 40[/tt]from the doku I guess the next numbers are:[tt]00 [/tt] the device ID[tt]02 [/tt] "WRITE_CMD"[tt]40 [/tt] means it's a BS memory, incrementing later on (0x41, 0x42...)[tt]00 01[/tt] address high and low byte[tt]00 02[/tt] counter high and low byte?Then my data would start with [tt]12[/tt], is this right?I'm a bit confused about the counter low byte which is [tt]7f[/tt] later on...I hope this question is not too silly, but my eyes start to flicker of the endless rows of numbers I looked at this evening Best,Michael Quote
TK. Posted February 22, 2008 Report Posted February 22, 2008 The term "counter" could be missleading, it actually means: SysEx block size / 8It's normaly the same for all blocks. 256 byte blocks are usually prefered, this results into a "counter value" of 0x100 >> 3 = 0x20Best Regards, Thorsten. Quote
audiocommander Posted February 24, 2008 Report Posted February 24, 2008 thanks for your answer, Thorsten.Unfortunately I'm still struggling with the byte-decoding. The syx2asm.pl script produces the correct output: the first byte is 0x04, which is fine, because it's the version compatibility info for v0.4 (see attached code snippets and ACSensorizer Patch Description) - however, if I use the suggested dec2bin/bin2dec method, I get a wrong result:convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.So in my case:[tt]0x12 = '0001 0010' ' 000 1001' = 0x09[/tt]but 0x09 is not the 0x04 I was hoping to for :'(I get the feeling, I'm missing something pretty obvious...Best Regards,Michael0: f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69 ...~@..@.....j2i16: 54 5e 30 1c 4c 06 00 00 00 00 00 00 00 0c 40 00 T^0.L.........@.32: 00 00 00 00 00 60 18 0c 06 03 01 00 00 00 00 00 .....`..........48: 00 02 02 01 41 00 50 30 1c 00 00 00 00 02 41 20 ....A.P0......A 64: 50 28 07 03 01 60 40 20 10 08 04 00 00 00 00 00 P(...`@ ........80: 00 00 00 00 00 00 00 00 00 10 08 04 02 00 20 10 .............. .96: 08 04 01 00 40 20 40 00 00 00 00 00 00 00 00 04 ....@ @.........(...)[/code][code=syx2asm.pl asm-output ----- this is correct]org 0x400000db 0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39 ;; (0x400000)db 0x30, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 ;; (0x400008)db 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x01 ;; (0x400010)db 0x81, 0x81, 0x81, 0x81, 0x00, 0x00, 0x00, 0x00 ;; (0x400018)db 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07 ;; (0x400020)db 0x00, 0x00, 0x00, 0x00, 0x14, 0x14, 0x14, 0x14 ;; (0x400028)db 0x07, 0x06, 0x07, 0x04, 0x04, 0x04, 0x04, 0x04 ;; (0x400030)and here the output from my calculations: 0: 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00 00 .j2iT^0.L....... 16: 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01 00 ..@......`...... 32: 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00 00 ........A.P0.... 48: 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04 00 ..A P(...`@ .... 64: 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08 04 ................ 80: 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00 00 .. .....@ @..... 96: 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02 01 ....f..F@...H...[/code][code=Binary representation] 0: 30 30 30 31 30 30 31 30 30 31 31 30 31 30 31 30 0001001001101010 16: 30 30 31 31 30 30 31 30 30 31 31 30 31 30 30 31 0011001001101001 32: 30 31 30 31 30 31 30 30 30 31 30 31 31 31 31 30 0101010001011110 48: 30 30 31 31 30 30 30 30 30 30 30 31 31 31 30 30 0011000000011100 64: 30 31 30 30 31 31 30 30 30 30 30 30 30 31 31 30 0100110000000110 80: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 96: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 0: 09 12 24 49 13 26 4d 1a 35 6a 54 28 51 23 46 0c ..$I.&M.5jT(Q#F. 16: 19 32 64 49 13 26 4d 1a 34 69 52 25 4a 15 2a 55 .2dI.&M.4iR%J.*U 32: 2a 54 28 51 22 45 0b 17 2f 5e 3c 78 71 63 46 0c *T(Q"E../^<xqcF. 48: 18 30 60 40 00 01 03 07 0e 1c 38 71 62 44 09 13 .0`@......8qbD.. 64: 26 4c 18 30 60 40 00 01 03 06 0c 18 30 60 40 00 &L.0`@......0`@. 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 96: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................[/code] Quote
TK. Posted February 24, 2008 Report Posted February 24, 2008 When you are starting at the 12th, and not at the 13th byte, it makes sense:First 8 bytes of BankStick content:0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39Binary representation:00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001Scrambled SysEx Stream:0x02 0x12 0x6a 0x32 0x69 0x54 0x5e 0x30 0x1c 0x4c 0x06Binary representation:0000010 0010010 1101010 0110010 1101001 1010100 1011110 0110000 0011100 1001100 0000110Now we concatenate the binary values to a single string (remove spaces)00000100010010110101001100101101001101010010111100110000001110010011000000110And split it into 8bit chunks:00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001 ...Convert it back into hex (or decimal) values:0x04 0x4b 0x53 0x2d 0x35 0x2f 0x30 0x39Surprise! ;)Best Regards, Thorsten. Quote
TK. Posted February 24, 2008 Report Posted February 24, 2008 When I read your initial question, I guess, that the 0x40 was confusing you.The 8th and 9th byte are the address, 0x4000 means: BankStick, offset 0Best Regards, Thorsten. Quote
Durisian Posted February 25, 2008 Author Report Posted February 25, 2008 I've had some success, but i've hit a brick wallI'm dealing with uploading/dowloading a whole 32k bankstickI've successfuly got the sysex message translated correctly to bankstick format.The initial method was skip the first 12 bytes, run the rest of the file through the algorithim. Then every 0x400 bytes I skip 12 bytes (i think it's was 12 anyway, might have been 8 or 10 :-[ my code is another computer).This worked great 100% accurate using a sysex file made with hex2syx.plGetting it back to a sysex message has proved much more difficult.So to convert to sysex. it's 11 bytes of header and info. followed by 0x400 bytes of bankstick data (in bankstick format). Which should be 0x492 bytes of sysex data.followed by the checksum and 0xF7Total 0x4A0 bytes.loop that 32 timesbut for some reason I fall 3 bytes short. Adding 3 blank bytes before the checksum almost worked. But my bankstick data ends up being offset +2 every 0x400 bytes. (using my sysex to bankstick code)So I go back to my sysex to bankstick conversion. To do it properly (hoping to find an answer). I take the header and info (11 bytes worth), convert the next 0x492 bytes (7-bit to 8-bit), get the checksum (to eventully compare it) and F7.Then loop the sequence 32 times to get the full bankstick.Taking the same sysex file I used earlier, The first loop works great, my first 0x400 bankstick bytes are correct, but as of the second loop the data is incorrect.I can relate to AC here... there has to be something obvious I'm missing.edit: added a bit more clarity Quote
stryd_one Posted February 25, 2008 Report Posted February 25, 2008 Why do I get the feeling, that your function is not adding the one bit, to convert 7b to 8b - either on the first or the last byte.... Just a gut feeling... Quote
audiocommander Posted February 25, 2008 Report Posted February 25, 2008 yay, it works!!thanks so much to all of you. I was making two errors: First, I have indeed been confused by the 0x40 (Bankstick address in the header), forgetting the low byte and therefore assumed to start with the 12th byte, but it's of course the 11th byte to begin with;secondly I misread the 0x20 (lenght of sysEx) for a 0x02 (12th byte)... typical kind of inattention-error, that's why I always failed in math at school ::)I'm going to make some more tests and then I will post some lines of C-code that may be useful for others (and probably for a Java conversion as well).Best,Michael...see the names in the third descrambled output? "KS-5/09" and "KS-5/SP" :D SysEx data output: 0: 02 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00 ..j2iT^0.L...... 16: 00 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01 ...@......`..... 32: 00 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00 .........A.P0... 48: 00 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04 ...A P(...`@ ... 64: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08 ................ 80: 04 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00 ... .....@ @.... 96: 00 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02 .....f..F@...H.. 112: 01 00 60 30 18 0c 05 0c 40 66 63 4f 7f 7f 7f 44 ..`0....@fcO...D 128: 16 4e 67 43 10 00 00 00 00 18 6d 66 23 31 7e 7f .NgC......mf#1~. 144: 3f 5f 60 44 5a 4c 5a 35 17 54 6a 03 01 40 04 02 ?_`DZLZ5.Tj..@.. 160: 00 40 20 10 23 10 00 00 00 00 00 00 58 50 02 01 .@ .#.......XP.. 176: 00 40 20 00 00 02 00 00 00 40 40 30 20 14 0c 00 .@ ......@@0 ... 192: 00 00 00 00 00 50 28 14 0a 01 00 20 28 0c 00 00 .....P(.... (... 208: 00 00 0d 16 23 6d 7e 7f 3f 5f 6f 70 00 08 00 00 ....#m~.?_op.... 224: 04 02 01 00 40 08 04 02 01 00 20 10 08 10 00 00 ....@..... ..... 240: 00 00 00 00 00 00 00 57 30 15 0a 20 00 00 00 52 .......W0.. ...R Binary 7-bit output: 0: 30 30 30 30 30 31 30 30 30 31 30 30 31 30 31 31 0000010001001011 16: 30 31 30 31 30 30 31 31 30 30 31 30 31 31 30 31 0101001100101101 32: 30 30 31 31 30 31 30 31 30 30 31 30 31 31 31 31 0011010100101111 48: 30 30 31 31 30 30 30 30 30 30 31 31 31 30 30 31 0011000000111001 64: 30 30 31 31 30 30 30 30 30 30 31 31 30 30 30 30 0011000000110000 80: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 96: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 112: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 128: 30 31 31 30 30 31 30 30 30 30 30 30 30 30 30 30 0110010000000000 144: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 160: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 176: 30 30 30 30 30 30 31 31 30 30 30 30 30 30 30 31 0000001100000001 192: 31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31 1000000110000001 208: 31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31 1000000110000001 224: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 240: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 0000000000000000 Converted 8-bit output: 0: 04 4b 53 2d 35 2f 30 39 30 30 00 00 00 00 00 00 .KS-5/0900...... 16: 64 00 00 00 00 00 03 01 81 81 81 81 00 00 00 00 d............... 32: 00 01 02 03 04 05 06 07 00 00 00 00 14 14 14 14 ................ 48: 07 06 07 04 04 04 04 04 00 00 00 00 00 00 00 00 ................ 64: 00 00 00 00 08 08 08 08 02 02 02 02 01 01 01 04 ................ 80: 00 00 00 00 00 00 00 00 99 8f 90 8d 00 00 00 52 ...............R 96: 02 02 02 03 03 03 03 02 8c 81 9b 1c ff ff ff c4 ................ 112: 2d 3b 3c 32 00 00 00 00 63 6e 64 6c 7f 7f 7f 7f -;<2....cndl.... 128: 04 4b 53 2d 35 2f 53 50 30 30 02 02 01 01 01 04 .KS-5/SP00...... 144: 64 00 00 00 00 00 0b 14 01 01 01 01 00 00 00 80 d............... 160: 00 01 02 03 04 05 06 00 00 00 00 00 14 14 14 14 ................ 176: 04 02 05 03 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f ........id{..... 192: 00 02 00 00 08 08 08 08 02 02 02 02 01 01 01 04 ................ 208: 00 00 00 00 00 00 00 00 57 60 54 52 00 00 00 52 ........W`TR...R 224: 02 02 02 02 03 03 03 02 8c 56 9b 53 ff ff ff c4 .........V.S.... 240: 23 27 14 13 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f #'......id{..... Quote
Durisian Posted February 26, 2008 Author Report Posted February 26, 2008 Nice work AC!!Have you had any success going back to sysex?I'm flat out at work for next 2 weeks. So I won't have time to work on it for a while. :( Quote
audiocommander Posted February 26, 2008 Report Posted February 26, 2008 nah, I'm not that far :)next step is to split up the numbers and feed them in a table. But it's the same here: I have an exam end of the week, so I can resume working on it next week soonest (and I better won't dare to overwrite my BS with cheesy data until I'm sure it works ;D )Best,Michael Quote
Durisian Posted March 1, 2008 Author Report Posted March 1, 2008 hmmm, I had brain wave last night... Can someone help me out with this...when converting 7bit to 8bit, it should be 1024 hex bytes to a section, times 32 to get a full bankstick.From reading a sysex file in a hex viewer, there is 1182 bytes to a section.So minus the header and info (11 bytes), minus footer (2 bytes).Leaves 1170 bytes that make up the bankstick (in 7bit).Given that 8 bytes of syx are converted to 7 bytes hex.1170 divided by 8 = 146.25146.25 * 7 = 1023.75what am I missing? ???Why do I get the feeling' date=' that your function is not adding the one bit, to convert 7b to 8b - either on the first or the last byte.... Just a gut feeling...[/quote']2 bits maybe? Quote
stryd_one Posted March 1, 2008 Report Posted March 1, 2008 Not sure about your algorithm, but 1182 - 11 - 2 = 1169 ;)Uhm... those fractions don't always work out, so make sure you round up to the nearest whole bit. No quantum PICs just yet :( Quote
audiocommander Posted March 1, 2008 Report Posted March 1, 2008 Hi Durisian,as I said, I'm not finished yet, but by using the approach Thorsten suggested, it seems to work so far.But remember: haven't tested enough and the checksum is totally ignored, so there is a 1-byte shift in the output; I only used it to read the BS contents – never in the other direction, to write back the edited data... normally I wouldn't post it like this ;)It's in C, the second snippet using some additional (easy to remove) Objective-C;the only C-lib-function used (IIRC) is strcat(), which is simply combining two strings to one (but note the end-byte [tt]\0[/tt])void dec2bin(int x, unsigned char *result) { int i; result[0] = 0; for (i = 7; i >= 0; i--) { strcat(result, x & (1 << i) ? "1": "0"); }}long bin2dec(unsigned char *s) { long r=0; for (; *s; r = (r<<1) | (*s++ - '0')); return r;}[/code]and this is the current decoding snippet; [tt]sxd[/tt] is an NSData Object (Objective-C), that is interchangeable with a binary SysEx-File. But before working on the bytes, I put them in a C-buffer, so conversion should be relatively easy:[code=SysEx decoder - early beta, use at your own risk]-(void)printAcd { // prints sensorizer patch data (sxd with stripped sysEx headers) if(sxd == nil) { // try to write data to file (for debug) // [sxd writeToFile:@"~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd" atomically:YES]; // try to read data from file (for debug) [self setSxd:[NSData dataWithContentsOfFile:@"~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd"]]; } unsigned long l = [sxd length]; unsigned long i,k; unsigned char buf[l]; [sxd getBytes:buf length:l]; for(i=0;i<l;i++) { switch(buf[i]) { case 0xf0: // skip header i += 11; break; case 0xf7: // skip EOX i++; break;// IMPORTANT: CHECKSUM NOT ATTENDED; THIS HAS TO BE DONE TO MAKE IT REALLY WORK } [acd appendBytes:&buf[i] length:sizeof(buf[i])]; } // debug out l = [acd length]; [acd getBytes:buf length:l];// printf("\n\nSysEx data output:\n");// hexview(buf, 256); // now we want to convert the bits and bytes from sysEx 7-bit back to the original 8-bit format l = [acd length]; char nbuf[l]; [acd getBytes:nbuf length:l]; unsigned char b[l*8]; // binaries string unsigned char a[l]; // 8-bit numbers array unsigned char s[8]; // 7-bit binary // build one long string of binary coded decimals for(i=0;i<l;i++) { dec2bin(nbuf[i],s); for(k=0;k<7;k++) { s[k] = s[k+1]; } s[7] = '\0'; strcat(b, s); } // debug out// printf("\n\nBinary 7-bit output:\n");// hexview(b, 256); // split b to 8-bit chunks unsigned char bchunk[9]; for(i=0;i<l;i++) { for(k=0;k<8;k++) { bchunk[k] = b[(i*8)+k]; } bchunk[8] = '\0'; a[i] = (unsigned char) (bin2dec(bchunk)); } // print hexview of b// printf("\n\nConverted 8-bit output:\n"); printf("\n8-bit ACS data, BS#%i\n", bankstick); hexview(a, l);}The conversion needs some time, but compared to the time the SysEx dump needs to arrive via MIDI, it's more than okay :)Hope this helps,best Michael Quote
Durisian Posted March 1, 2008 Author Report Posted March 1, 2008 stryd: on the long drive home from work I realised that 1184 - 11 -2 is actually 1171..........typo + bad maths = stop thinking about midibox and get more sleep ;DI was trying to point out that I had obviously missed something because there wasn't enough bits in the sysex message by my calculations. As it turns out there is 5 to many, I would say this is where my algorithm (for both directions) goes wrong, it doesn't account for these extra bits.This is what i've been missingI didn't pick it up earlier because I was using mbasecalc for the math, where anything after the decimal point is ignored :-[. And also by cutting corners by only dealing a whole bankstick, rather than making it properly mios compatiableHey wow, thanks AC.Now all I need is some time when my brain is switched on Quote
Durisian Posted March 13, 2008 Author Report Posted March 13, 2008 yay, it works.Now onto transmit and recieve over the midi ports... Quote
wackazong Posted September 11, 2008 Report Posted September 11, 2008 Hey Audiocommander,have you developed your C routines any further? I want to send some data to my core via sysex, and would love to not have to write my own scramble/descramble routine.Best, ALEXander. Quote
audiocommander Posted September 11, 2008 Report Posted September 11, 2008 Hi ALEXander,unfortunately no :-(I would love to, but there are too many things I have to do the next semester. I hope that I can resume workin on this when I will finally update the ACSensorizer to the new project structure (which will be either in winter-holidays or in march 09 latest (oh, how I'm looking forward to march when my jobs will be finished :-)but I would be very interested if you could keep us updated !Cheers,Michael Quote
TK. Posted September 11, 2008 Report Posted September 11, 2008 Scrambling is not required, you can also use these routines, which are easier to handle:http://svnmios.midibox.org/listing.php?repname=svn.mios&path=%2Ftrunk%2Fapps%2Fexamples%2Fsyx_dump%2FDetails are described in the README.txt fileBest Regards, Thorsten. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.