Programming Bankstick Files

[robinfawell]

Although this query relates to my organ project, this is likely to be of general interest.

QUERY

I believe that I can write sysex files to a Bankstick using the MIOS Sysex implemetation commands, using command 03 and then Command 02.

To test that the files are stored I would then, using Commands 03 and 01, read from the Bankstick.

I would use MIDIOX to send and receive the data files.

Is this procedure correct?

Do I need to prepare the Core module other than giving it an ID and programming the bootstraps, MIOS and Midio128?

I do realise that ultimately the send routines to transmit the sysex messages to my sound module will be controlled by a button handler.

Regards Robin Fawell

[TK.]

Yes, in principle thats correct.

There is a tutorial application in the MIOS download section which explains BankStick transfers step-by-step, search for "bs_xfer_v1_4.zip"

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

Thank you for the last reply.

You referred me to the BS Transfer Method.  In Step 1 you state that the file Main.asm includes P=EEPROM16.  It contains "LIST      P=PIC18F452, R=DEC".  There is no reference to EEPROM16.

I am not ready yet to modify the main.asm file and it seems easier (for me) to use the MIOS Sysex implementation method.

It might be better if I explained in detail what I am trying to do.

I have recorded using Midiox several sysex messages.  These are "setup" instructions for the Roland SC8850 Sound Module.  I want to store these in my Bankstick module (0-7). This is built but not tested.

Checksum

In the sysex implementation text page 3, it refers to "checksum: inverted  checksum from ah to the last byte of <dump>.

In hex to Sysex converter you refer to "-(sum of bytes) & )x7f.  

Are we counting 8 bit or 16 bit bytes?

If we count the bytes fom ah to the last dump byte in binary invert the result ie 0101 becomes 1010 and then "AND" the result with 7f (0111 1111) would this be correct?

Do we always have to use perl to convert to syx from hex to send midi messages via Midi in?

Thanks Robin

[robinfawell]

I left this out of my preceding message.

In checking through the Sysex Implementation text under the command 01 examples I think there may be an error in the sysex message for the Internal Flash..  I believe that the ch parameter should be 0x20.

If Counter = 0x08000 When this is shifted 10  and "AND"ed with 7f the result is 0x20  (binary 0010 0000) not 0x10.  Of course I may be wrong and misunderstood the basis of the calculation.

Regards Robin

[TK.]

Hi Robin,

you are right, I forgot to update the documentation about the EEPROM16 type, it must be PIC18F452 - EEPROM16 doesn't work due to an imperfection in the MPASM assembler

you are also right with the documentation error concerning the internal flash readout counter - ch must be 0x20

But regarding the SysEx upload: it isn't possible to write any plain MIDI data (like SysEx strings) into the BankStick, they have to be encoded by the hex2syx script to another format. It's too complicated to convert an existing data dump into the MIOS upload format without this script.

So, what you have to do: you have to write a script which converts your binary files to assembler text, so that the assembler can generate a .hex file, which can be thereafter converted to a .syx file by hex2syx.pl

The assembler format can be found in bankstick_content.inc

Best Regards, Thorsten.

[TK.]

Hi again,

I've updated the bs_xfer package (MIOS Download -> bs_xfer_v1_6.zip)

It now also contains a script which allows you to convert a plain binary file to assembler text, it can be found in the tools directory (tools\bin2asm.pl)

So, here the flow to bring plain binaries into the BankStick:

• use bin2asm.pl to convert your binary file to assembler
  
• you can copy&paste several assembler files to a single assembler file. Only requirement: the overall number of bytes should not exceed the 32768 boundary (0x407fff) if you are using a 32k BankStick, or 65536 (0x40ffff) if you are using a 64k BankStick
  
• use MPASM to get a .hex file
  
• use hex2syx.pl to get a MIOS upload file
  

You will notice that this conversion process simplifies the creation of the BankStick content a lot, since you are able to move and rearrange the converted binary files with a text editor to free BankStick addresses

Best Regards, Thorsten.

[robinfawell]

Thanks Thorsten for such a fast response.

May I ask you to clarify the checksum queries.

Regards Robin

[TK.]

Hi Robin,

I thought that the checksum question became obsolete since you will use the hex2syx.pl script which calculates the checksum automatically.

Maybe you've misinterpreted the term "sum of bytes". It should mean: the checksum is the addition result of all bytes within the address/counter field and all bytes of the dump field. Than it will be negated, so that the checksum within the SysEx file + the calculated checksum (which will be calculated by MIOS during the upload) results to zero

The checksum is a 7-bit value, therefore only the first 7 bits are used (thats the "& 0x7f")

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

Thanks for straightening me out.

I was in the middle of assembling a binary file complete with 01, 03 commands.

I am going to practice with my largest file, "reset" which will go to BS 00.

I will generate a reset.inc file using bin2asm.pl.

I will use MPASM to generate a .hex file.

I will convert this to a syx file using hex2syx.pl and then upload this using Midiox.

MPASM

Do I use the default settings?

I assume the source file is Reset.inc

Processor EEPROM or 18F452?

Which generated file?

Tab size 8 ?

Thanks again Robin

[TK.]

Hi Robin,

I would suggest to create a copy of the main.asm which which comes with the bs_xfer package, because it already contains all required settings (and therefore documentates the requirements and answers your questions)

From main.asm you can include one or more .inc files which contain the converted binaries.

Hint: after the assembler has been passed, you will find the address offsets to the converted binaries in main.lst - they are starting at 0x400000, the last 4 digits (16 bits) are equal to the BankStick addresses.

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

I have received the following response from the core module via Midi out after sending a test  .syx file.

F0 00 00 7E 40 00 0F 07 F7

F0 00 00 7E 40 00 0F 7D F7

F0 00 00 7E 40 00 0F 3A F7

F0 00 00 7E 40 00 0F 6E F7

F0 00 00 7E 40 00 01 F7

The file was sent to BS 7.  I have worked out that the messages are to do with WREG.

The last is a request for an upload, I think.  

Does this message seem OK?

Regards Robin

PS device ID is 0x00

[TK.]

Hi Robin,

yes, the whole upload procedure was successfull:

>  F0 00 00 7E 40 00 0F 07 F7  

is an acknowledge message (0F) which reports that the received checksum matched with the calculated checksum, that the data has been copied into the BankStick and that it has been verified (read-back and compare)

The value behind 0F (here: 07) is the same checksum like reported by the hex2syx.pl script.

You will receive an error message (0E) if any step fails (e.g. BankStick not connected, checksum error, verify error, etc...)

>  F0 00 00 7E 40 00 01 F7

is the upload request of the first level bootstrap loader.

A reset instruction will be issued after up- or downloads via MIOS, therefore this message appears like on a common startup

So - now you are sure that the binary has been copied into the BankStick.

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

I am progressing slowly but have reached another difficulty.

So far I have uploaded a large and medium converted binary file to BS 0 and 7.

The large file "Reset"  bin file was recorded as 16470 bytes by Midiox.  After conversion to Reset.syx the file size measured by Midiox was 59025 bytes.

During the upload Midiox received 51 Sysex acknowledgements. .

For the medium file the respective bytes were 960 and 3560 with 5 ack.

As you stated in your last message it seems as if data is being stored.

I have tried many different ways to read the stored data using the sysex read procedures with Midiox with no success.  Is it possible to use the sysex Read 01 Command for this task?

I am sorry to trouble you again so soon.

Regards Robin

[TK.]

Hi Robin,

it's normal that the resulting file is larger than the original binary after the hex2syx conversion -- this is due to the required conversion algorithm which converts 8-bit values to 7-bit values. MIOS converts them back to the 8-bit values during the upload.

So, normaly you should get about 1184 bytes per 1k block (original size: 1024 bytes)

But it's very strange that the converted .syx files are much larger.

How did you convert the binary exactly?

What is the content of the resulting assembler file? You should see the address offsets within the files (they are inserted as comments at the end of each line), what is the last offset value?

Best Regards, Thorsten.

[robinfawell]

Thanks Thorsten

I recorded the sysex files using MIDIOX.

I saved the data from MIDIOX as a .txt file.

I created an .asm file from the .txt file using bin2asm.

Using MPLAB I made an .hex file using the .asm file as source file. Note this is the only file I used.

I converted this file to a .syx file using hex2syx.

I used MIDIOX to send the data to the Core via Midi in.

The last offset on the "Reset" fle is 0x40C0A0.  There is a change in the format up to 0x4080B0.  The data up to that point was db followed by eight double bytes ;; and then the address eg 40000 (1st add).

After this there are numerous db data sets per line.

As if the line return had been lost.

Regards Robin

[TK.]

Hi Robin,

I saved the data from MIDIOX as a .txt file.

This could be the reason why the converted .syx file is more than 3 times larger than the original file.

You've to save the SysEx data in binary format - just like a common .syx file. Thereafter use bin2asm.pl to convert this binary to a assembler file.

Check the content of the generated assembler code, the data behind the db statements must match with the data displayed in the MIDI-Ox window.

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

Following your guidance I have produced a new reset.syx file.  Originally as recorded with 16470 bytes after conversion to a syx file now 20137. (both measured by MIDIOX.

By the way with MPLAB I did not use main.asm with a Reset.inc file.  I made a reset.asm and used it (only) as a source file.  Was this OK?

Can I use Command 01 to read the BS?  As you probably realise I keep thinking that the MIOS system will respond to Midi sysex messages on Midi In.  Plese let me know how to proceed.

Thanks Robin

[TK.]

Hi Robin,

alright - the size of 20137 sounds correct, these are 17 converted 1k blocks, and the original file was a little bit larger than 16k - perfect!

Yes, now you could read out the BankStick again and convert the resulting .syx to a .asm file like described in the bs_xfer package. However, it's more like a toy, but it helps you to 100% ensure that the file has been uploaded.

For a first test it's ok when you assemble the reset.asm file directly, but later you propably want to have multiple files in a single BankStick, and for this it would be easier to include all data files into a single assembler file, so that the assembler can take care about the data locations.

Ok, now you should proceed with the function which sends the SysEx

string, see:

http://www.midibox.org/cgi-bin/yabb/YaBB.cgi?board=mios;action=display;num=1078174868

Later you should think about how to organize the data. I would suggest

to create a table which contains:

• the bankstick number
  
• the offset address
  
• the size of the SysEx string

But I fear that I don't have enough time to explain more details before my holidays... :-/

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

I can see the point of having multiple sysex strings held in each bankstick and being uploaded in one operation.

I have tried many times using MPlab to include the .inc files into the.asm file without any result.  The build is signalled as successful but I finish up with the original .asm file.

I note that each file (.asm or .inc ) starts with Address 400000.   Should not the .inc files  start with  different address(es)?

There are also comments in each .inc file re deleting the line where there is a LIST instruction.  Should these be followed?

Please confirm that I should be using MPLAB to make the composite .asm file and advise  any other instruction needed.

Regards Robin

Since sending the above message I wondered whether I should be composing the composite "write to bankstick" by cutting and pasting within MIDIOX.

[TK.]

Hi Robin,

did you solve the problem in the meantime?

Root cause: overlapping data areas.

You've to remove the "org" statements and to add a single one (which points to 0x400000) to the main.asm, or you have to relocate the data blocks by hand to free addresses.

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

I have removed the org statements from the .inc files and placed a .org statement in the main.asm file.  I have removed the LIST statement re PIC 18F 452 and the END from all three .inc files.  I showed 3 include statements (for reset, trems and tremf) in the main.asm and removed the bankstick content  .inc line.

I have produced a main.lst file which to me looks promising.  The reset .inc was 16k+ bytes and the trems and tremf .inc file were both 2.5k+ bytes.  The main.lst file lists the three files however the list still shows the three files all originating from address 400000  admittedly  with ;; in front.

The problem is that when I try to convert the main.hex fle to main.syx I get no result other than a line showing the directory.  (There is a main.syx file produced but it is blank)  I have used the same procedure with a much smaller .hex file and it seems to work.

I should add that the main.hex file is about 65k bytes.

Please help.

Regards Robin

[TK.]

Hi,

just send me your project directory so that I can search for the reason.

Best Regards, Thorsten.

[robinfawell]

Dear Thorsten

I have sent you 2 emails. One has an attachment with the project diretory file.

The 2nd contains an outline of the organ project.

Regards Robin

[robinfawell]

Thanks Thorsten,

I have been on holiday for the last week or so.  The latest main.lst looks correct.

I am still unsure whether the MIOS system will send the individual files stored in the BS to the Sound Module correctly.

If I use the procedure outlined in http://www.midibox.org/cgi-bin/yabb/YaBB.cgi?board=mios;action=display;num=1078174868

Will I get the file transmitted as I originally recorded by Midiox ?

I know that using the Sysex instructions via Midiox produces altered files and increases the file size..  In the tutorial you state that a 32k BS produces 39k file size.

Please comment.

Regards Robin

[TK.]

Hi Robin,

somewhere is a misunderstanding, but I don't know where exactly, so I just try to explain the transfer flow again in short words:

• you've a binary file, let's say abc.syx
  
• with bin2asm.pl you convert it to assembler lines, several binaries can be combined to a single assembler program in this way
  
• with MPASM the code will be assembled to a single data dump, the result is a .hex file
  
• with hex2syx.pl you scramble the data dump into a MIOS specific and especially MIDI compatible format
  
• MIOS descrambles the data stream back to the original binary format during the upload

After the transfer the transfered memory content is the same like specified in the .hex file

You could read back the data and descramble it with syx2asm.pl to check the BankStick content. This procedure is explained in the bs_xfer example...

Why it is required to scramble/descramble the data during the MIOS upload? Because the binary data contains 8-bit values, but the MIDI spec only allows to embedd 7-bit values into a SysEx stream. Therefore hex2syx.pl rearranges the data so that no bit will get lost (you will notice that all values within a SysEx stream are between 0x00 and 0x7f, and never >= 0x80) and MIOS rearranges the bits again to reconstruct the original 8-bit values.

Without this (clever) mechanism it wouldn't be possible to upload any data, not only the BankStick content, but also the flash and EEPROM content. Therefore it's proven.

Best Regards, Thorsten.