Jump to content

7-bit sysex scrambling from mios


Durisian
 Share

Recommended Posts

So I'm experimenting with developing a PC editor for my pedal board project. At the moment i'm just reading a syx file downloaded from a bankstick

I'm hoping someone can enlighten me as to the 7-bit scrambling.

I have assumed it's the first byte of 8 that contains the first bit of the following 7.

0ABCDEFG, 0AAAAAAA, 0BBBBBBB, 0CCCCCCC, 0DDDDDDDD, 0EEEEEEE, 0FFFFFFF, 0GGGGGGG.

So to get the value of Bankstick address 0x0000

Given that the header is 11 bytes long, it should be:

value = ((12thByte & b01000000) << 1) | 13thByte

Can anyon help me out?

Cheers!

Link to comment
Share on other sites

No, this kind of scrambling isn't used by MIOS, instead the bits are packed without de-positioning.

You will find two useful perl scripts, which include the algorithms, in the MIOS source code package under tools/hex2syx.pl and tools/syx2asm.pl

Best Regards, Thorsten.

Link to comment
Share on other sites

There is a Java based scrambling routine in the MIOS Studio package - but it doesn't descramble.

Unfortunately Java is not like e, where you can easily define a bitvector with unlimited length, pack the 8bit value array into this vector, and unpack a 7bit value array from the vector. Thats basically the approach. In "e" a single line of code, in Java/C... hm... perhaps some more.

Best Regards, Thorsten.

Link to comment
Share on other sites

With Java (where memory consumption doesn't matter), you could maybe also do it the following way: convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.

Pseudo code:

  string := ""

  foreach byte

     string := string + decimal_to_binary(byte)

  while lenth(string) && (length(string) >= 7) {

    SendMIDI(binary_to_decimal(string[0:6]))

    string := string[7:end]

  }

(this is no official programming language, I only want to demonstrate the algorithm...)

Best Regards, Thorsten.

Link to comment
Share on other sites

  • 3 weeks later...

Hello number-crunching friends!

I'm also trying to figure that out for a patch editor written in C;

however, I am not sure about the length of the returned sysEx header and would appreciate some clarification.

I get these (hex-) values when I request the first bankstick's content:

[tt]f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69[/tt]

from the syx2asm.pl I see the header is:

[tt]f0 00 00 7e 40[/tt]

from the doku I guess the next numbers are:

[tt]00  [/tt] the device ID

[tt]02  [/tt] "WRITE_CMD"

[tt]40  [/tt] means it's a BS memory, incrementing later on (0x41, 0x42...)

[tt]00 01[/tt] address high and low byte

[tt]00 02[/tt] counter high and low byte?

Then my data would start with [tt]12[/tt], is this right?

I'm a bit confused about the counter low byte which is [tt]7f[/tt] later on...

I hope this question is not too silly, but my eyes start to flicker of the endless rows of numbers I looked at this evening freak.gif

Best,

Michael

Link to comment
Share on other sites

thanks for your answer, Thorsten.

Unfortunately I'm still struggling with the byte-decoding.

The syx2asm.pl script produces the correct output: the first byte is 0x04, which is fine, because it's the version compatibility info for v0.4 (see attached code snippets and ACSensorizer Patch Description)

- however, if I use the suggested dec2bin/bin2dec method, I get a wrong result:

convert the decimal value of each 8bit byte into binary strings, and concatenate them one after another to a long, new string. Thereafter split the string into chunks of 7 characters, and convert back binary->decimal to get the 7bit value.

So in my case:[tt]

0x12 = '0001 0010'

      ' 000 1001' = 0x09[/tt]

but 0x09 is not the 0x04 I was hoping to for  :'(

I get the feeling, I'm missing something pretty obvious...

Best Regards,

Michael



0: f0 00 00 7e 40 00 02 40 00 01 00 02 12 6a 32 69  ...~@..@.....j2i
16: 54 5e 30 1c 4c 06 00 00 00 00 00 00 00 0c 40 00  T^0.L.........@.
32: 00 00 00 00 00 60 18 0c 06 03 01 00 00 00 00 00  .....`..........
48: 00 02 02 01 41 00 50 30 1c 00 00 00 00 02 41 20  ....A.P0......A
64: 50 28 07 03 01 60 40 20 10 08 04 00 00 00 00 00  P(...`@ ........
80: 00 00 00 00 00 00 00 00 00 10 08 04 02 00 20 10  .............. .
96: 08 04 01 00 40 20 40 00 00 00 00 00 00 00 00 04  ....@ @.........
(...)
[/code]

[code=syx2asm.pl asm-output ----- this is correct]org 0x400000
db 0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39   ;; (0x400000)
db 0x30, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00   ;; (0x400008)
db 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x01   ;; (0x400010)
db 0x81, 0x81, 0x81, 0x81, 0x00, 0x00, 0x00, 0x00   ;; (0x400018)
db 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07   ;; (0x400020)
db 0x00, 0x00, 0x00, 0x00, 0x14, 0x14, 0x14, 0x14   ;; (0x400028)
db 0x07, 0x06, 0x07, 0x04, 0x04, 0x04, 0x04, 0x04   ;; (0x400030)

and here the output from my calculations:


    0: 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00 00  .j2iT^0.L.......
   16: 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01 00  ..@......`......
   32: 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00 00  ........A.P0....
   48: 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04 00  ..A P(...`@ ....
   64: 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08 04  ................
   80: 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00 00  .. .....@ @.....
   96: 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02 01  ....f..F@...H...
[/code]

[code=Binary representation]
    0: 30 30 30 31 30 30 31 30 30 31 31 30 31 30 31 30  0001001001101010
   16: 30 30 31 31 30 30 31 30 30 31 31 30 31 30 30 31  0011001001101001
   32: 30 31 30 31 30 31 30 30 30 31 30 31 31 31 31 30  0101010001011110
   48: 30 30 31 31 30 30 30 30 30 30 30 31 31 31 30 30  0011000000011100
   64: 30 31 30 30 31 31 30 30 30 30 30 30 30 31 31 30  0100110000000110
   80: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
   96: 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000


    0: 09 12 24 49 13 26 4d 1a 35 6a 54 28 51 23 46 0c  ..$I.&M.5jT(Q#F.
   16: 19 32 64 49 13 26 4d 1a 34 69 52 25 4a 15 2a 55  .2dI.&M.4iR%J.*U
   32: 2a 54 28 51 22 45 0b 17 2f 5e 3c 78 71 63 46 0c  *T(Q"E../^<xqcF.
   48: 18 30 60 40 00 01 03 07 0e 1c 38 71 62 44 09 13  .0`@......8qbD..
   64: 26 4c 18 30 60 40 00 01 03 06 0c 18 30 60 40 00  &L.0`@......0`@.
   80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
   96: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[/code]

Link to comment
Share on other sites

When you are starting at the 12th, and not at the 13th byte, it makes sense:

First 8 bytes of BankStick content:

0x04, 0x4b, 0x53, 0x2d, 0x35, 0x2f, 0x30, 0x39

Binary representation:

00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001

Scrambled SysEx Stream:

0x02 0x12 0x6a 0x32 0x69 0x54 0x5e 0x30 0x1c 0x4c 0x06

Binary representation:

0000010 0010010 1101010 0110010 1101001 1010100 1011110 0110000 0011100 1001100 0000110

Now we concatenate the binary values to a single string (remove spaces)

00000100010010110101001100101101001101010010111100110000001110010011000000110

And split it into 8bit chunks:

00000100 01001011 01010011 00101101 00110101 00101111 00110000 00111001 ...

Convert it back into hex (or decimal) values:

0x04 0x4b 0x53 0x2d 0x35 0x2f 0x30 0x39

Surprise! ;)

Best Regards, Thorsten.

Link to comment
Share on other sites

I've had some success, but i've hit a brick wall

I'm dealing with uploading/dowloading a whole 32k bankstick

I've successfuly got the sysex message translated correctly to bankstick format.

The initial method was skip the first 12 bytes, run the rest of the file through the algorithim. Then every 0x400 bytes I skip 12 bytes (i think it's was 12 anyway, might have been 8 or 10 :-[ my code is another computer).

This worked great 100% accurate using a sysex file made with hex2syx.pl

Getting it back to a sysex message has proved much more difficult.

So to convert to sysex. it's 11 bytes of header and info.

followed by 0x400 bytes of bankstick data (in bankstick format). Which should be 0x492 bytes of sysex data.

followed by the checksum and 0xF7

Total 0x4A0 bytes.

loop that 32 times

but for some reason I fall 3 bytes short. Adding 3 blank bytes  before the checksum almost worked. But my bankstick data ends up being offset +2 every 0x400 bytes. (using my sysex to bankstick code)

So I go back to my sysex to bankstick conversion. To do it properly (hoping to find an answer).

I take the header and info (11 bytes worth), convert the next 0x492 bytes (7-bit to 8-bit), get the checksum (to eventully compare it) and F7.

Then loop the sequence 32 times to get the full bankstick.

Taking the same sysex file I used earlier, The first loop works great, my first 0x400 bankstick bytes are correct, but as of the second loop the data is incorrect.

I can relate to AC here... there has to be something obvious I'm missing.

edit: added a bit more clarity

Link to comment
Share on other sites

yay, it works!!

thanks so much to all of you.

I was making two errors: First, I have indeed been confused by the 0x40 (Bankstick address in the header), forgetting the low byte and therefore assumed to start with the 12th byte, but it's of course the 11th byte to begin with;

secondly I misread the 0x20 (lenght of sysEx) for a 0x02 (12th byte)... typical kind of inattention-error, that's why I always failed in math at school ::)

I'm going to make some more tests and then I will post some lines of C-code that may be useful for others (and probably for a Java conversion as well).

Best,

Michael

...see the names in the third descrambled output? "KS-5/09" and "KS-5/SP" :D

SysEx data output:
    0:		02 12 6a 32 69 54 5e 30 1c 4c 06 00 00 00 00 00  ..j2iT^0.L......
   16:		00 00 0c 40 00 00 00 00 00 00 60 18 0c 06 03 01  ...@......`.....
   32:		00 00 00 00 00 00 02 02 01 41 00 50 30 1c 00 00  .........A.P0...
   48:		00 00 02 41 20 50 28 07 03 01 60 40 20 10 08 04  ...A P(...`@ ...
   64:		00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 08  ................
   80:		04 02 00 20 10 08 04 01 00 40 20 40 00 00 00 00  ... .....@ @....
   96:		00 00 00 00 04 66 1f 10 46 40 00 00 02 48 04 02  .....f..F@...H..
  112:		01 00 60 30 18 0c 05 0c 40 66 63 4f 7f 7f 7f 44  ..`0....@fcO...D
  128:		16 4e 67 43 10 00 00 00 00 18 6d 66 23 31 7e 7f  .NgC......mf#1~.
  144:		3f 5f 60 44 5a 4c 5a 35 17 54 6a 03 01 40 04 02  ?_`DZLZ5.Tj..@..
  160:		00 40 20 10 23 10 00 00 00 00 00 00 58 50 02 01  .@ .#.......XP..
  176:		00 40 20 00 00 02 00 00 00 40 40 30 20 14 0c 00  .@ ......@@0 ...
  192:		00 00 00 00 00 50 28 14 0a 01 00 20 28 0c 00 00  .....P(.... (...
  208:		00 00 0d 16 23 6d 7e 7f 3f 5f 6f 70 00 08 00 00  ....#m~.?_op....
  224:		04 02 01 00 40 08 04 02 01 00 20 10 08 10 00 00  ....@..... .....
  240:		00 00 00 00 00 00 00 57 30 15 0a 20 00 00 00 52  .......W0.. ...R


Binary 7-bit output:
    0:		30 30 30 30 30 31 30 30 30 31 30 30 31 30 31 31  0000010001001011
   16:		30 31 30 31 30 30 31 31 30 30 31 30 31 31 30 31  0101001100101101
   32:		30 30 31 31 30 31 30 31 30 30 31 30 31 31 31 31  0011010100101111
   48:		30 30 31 31 30 30 30 30 30 30 31 31 31 30 30 31  0011000000111001
   64:		30 30 31 31 30 30 30 30 30 30 31 31 30 30 30 30  0011000000110000
   80:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
   96:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  112:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  128:		30 31 31 30 30 31 30 30 30 30 30 30 30 30 30 30  0110010000000000
  144:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  160:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  176:		30 30 30 30 30 30 31 31 30 30 30 30 30 30 30 31  0000001100000001
  192:		31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31  1000000110000001
  208:		31 30 30 30 30 30 30 31 31 30 30 30 30 30 30 31  1000000110000001
  224:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000
  240:		30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30  0000000000000000


Converted 8-bit output:
    0:		04 4b 53 2d 35 2f 30 39 30 30 00 00 00 00 00 00  .KS-5/0900......
   16:		64 00 00 00 00 00 03 01 81 81 81 81 00 00 00 00  d...............
   32:		00 01 02 03 04 05 06 07 00 00 00 00 14 14 14 14  ................
   48:		07 06 07 04 04 04 04 04 00 00 00 00 00 00 00 00  ................
   64:		00 00 00 00 08 08 08 08 02 02 02 02 01 01 01 04  ................
   80:		00 00 00 00 00 00 00 00 99 8f 90 8d 00 00 00 52  ...............R
   96:		02 02 02 03 03 03 03 02 8c 81 9b 1c ff ff ff c4  ................
  112:		2d 3b 3c 32 00 00 00 00 63 6e 64 6c 7f 7f 7f 7f  -;<2....cndl....
  128:		04 4b 53 2d 35 2f 53 50 30 30 02 02 01 01 01 04  .KS-5/SP00......
  144:		64 00 00 00 00 00 0b 14 01 01 01 01 00 00 00 80  d...............
  160:		00 01 02 03 04 05 06 00 00 00 00 00 14 14 14 14  ................
  176:		04 02 05 03 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f  ........id{.....
  192:		00 02 00 00 08 08 08 08 02 02 02 02 01 01 01 04  ................
  208:		00 00 00 00 00 00 00 00 57 60 54 52 00 00 00 52  ........W`TR...R
  224:		02 02 02 02 03 03 03 02 8c 56 9b 53 ff ff ff c4  .........V.S....
  240:		23 27 14 13 00 00 00 00 69 64 7b 7f 7f 7f 7f 7f  #'......id{.....

Link to comment
Share on other sites

nah, I'm not that far  :)

next step is to split up the numbers and feed them in a table.

But it's the same here: I have an exam end of the week, so I can resume working on it next week soonest (and I better won't dare to overwrite my BS with cheesy data until I'm sure it works ;D )

Best,

Michael

Link to comment
Share on other sites

hmmm, I had brain wave last night... Can someone help me out with this...

when converting 7bit to 8bit, it should be 1024 hex bytes to a section, times 32 to get a full bankstick.

From reading a sysex file in a hex viewer, there is 1182 bytes to a section.

So minus the header and info (11 bytes), minus footer (2 bytes).

Leaves 1170 bytes that make up the bankstick (in 7bit).

Given that 8 bytes of syx are converted to 7 bytes hex.

1170 divided by 8 = 146.25

146.25 * 7 = 1023.75

what am I missing?   ???

Why do I get the feeling' date=' that your function is not adding the one bit, to convert 7b to 8b - either on the first or the last byte.... Just a gut feeling...

[/quote']

2 bits maybe?

Link to comment
Share on other sites

Hi Durisian,

as I said, I'm not finished yet, but by using the approach Thorsten suggested, it seems to work so far.

But remember: haven't tested enough and the checksum is totally ignored, so there is a 1-byte shift in the output; I only used it to read the BS contents – never in the other direction, to write back the edited data... normally I wouldn't post it like this ;)

It's in C, the second snippet using some additional (easy to remove) Objective-C;

the only C-lib-function used (IIRC) is strcat(), which is simply combining two strings to one (but note the end-byte [tt]\0[/tt])


void dec2bin(int x, unsigned char *result) {
int i;
result[0] = 0;
for (i = 7; i >= 0; i--) {
strcat(result, x & (1 << i) ? "1": "0");
}
}

long bin2dec(unsigned char *s) {
long r=0;
for (; *s; r = (r<<1) | (*s++ - '0'));
return r;
}
[/code]

and this is the current decoding snippet; [tt]sxd[/tt] is an NSData Object (Objective-C), that is interchangeable with a binary SysEx-File. But before working on the bytes, I put them in a C-buffer, so conversion should be relatively easy:

[code=SysEx decoder - early beta, use at your own risk]
-(void)printAcd {
// prints sensorizer patch data (sxd with stripped sysEx headers)

if(sxd == nil) {
// try to write data to file (for debug)
// [sxd writeToFile:@"~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd" atomically:YES];
// try to read data from file (for debug)
[self setSxd:[NSData dataWithContentsOfFile:@"~/ACSensorizer_PatchManager/BS_dir/ACS_BS_NSData.acd"]];
}

unsigned long l = [sxd length];
unsigned long i,k;
unsigned char buf[l];
[sxd getBytes:buf length:l];
for(i=0;i<l;i++) {
switch(buf[i]) {
case 0xf0: // skip header
i += 11;
break;
case 0xf7: // skip EOX
i++;
break;
// IMPORTANT: CHECKSUM NOT ATTENDED; THIS HAS TO BE DONE TO MAKE IT REALLY WORK
}
[acd appendBytes:&buf[i] length:sizeof(buf[i])];
}
// debug out
l = [acd length];
[acd getBytes:buf length:l];
// printf("\n\nSysEx data output:\n");
// hexview(buf, 256);

// now we want to convert the bits and bytes from sysEx 7-bit back to the original 8-bit format
l = [acd length];
char nbuf[l];
[acd getBytes:nbuf length:l];
unsigned char b[l*8]; // binaries string
unsigned char a[l]; // 8-bit numbers array
unsigned char s[8]; // 7-bit binary
// build one long string of binary coded decimals
for(i=0;i<l;i++) {
dec2bin(nbuf[i],s);
for(k=0;k<7;k++) {
s[k] = s[k+1];
}
s[7] = '\0';
strcat(b, s);
}
// debug out
// printf("\n\nBinary 7-bit output:\n");
// hexview(b, 256);

// split b to 8-bit chunks
unsigned char bchunk[9];
for(i=0;i<l;i++) {
for(k=0;k<8;k++) {
bchunk[k] = b[(i*8)+k];
}
bchunk[8] = '\0';
a[i] = (unsigned char) (bin2dec(bchunk));
}
// print hexview of b
// printf("\n\nConverted 8-bit output:\n");
printf("\n8-bit ACS data, BS#%i\n", bankstick);
hexview(a, l);

}

The conversion needs some time, but compared to the time the SysEx dump needs to arrive via MIDI, it's more than okay :)

Hope this helps,

best

Michael

Link to comment
Share on other sites

stryd: on the long drive home from work I realised that 1184 - 11 -2 is actually 1171....

...

...

typo + bad maths = stop thinking about midibox and get more sleep  ;D

I was trying to point out that I had obviously missed something because there wasn't enough bits in the sysex message by my calculations. As it turns out there is 5 to many, I would say this is where my algorithm (for both directions) goes wrong, it doesn't account for these extra bits.

This is what i've been missing

I didn't pick it up earlier because I was using mbasecalc for the math, where anything after the decimal point is ignored  :-[.  And also by cutting corners by only dealing a whole bankstick, rather than making it properly mios compatiable

Hey wow, thanks AC.

Now all I need is some time when my brain is switched on

Link to comment
Share on other sites

  • 2 weeks later...
  • 5 months later...

Hi ALEXander,

unfortunately no :-(

I would love to, but there are too many things I have to do the next semester. I hope that I can resume workin on this when I will finally update the ACSensorizer to the new project structure (which will be either in winter-holidays or in march 09 latest (oh, how I'm looking forward to march when my jobs will be finished :-)

but I would be very interested if you could keep us updated !

Cheers,

Michael

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
 Share

×
×
  • Create New...