M4 macro processing language for NG files

[Duggle]

Motivation

It is possible to compile from source files for NGC, NGL and NGR rather than type them directly.

The idea here is motivated by my rather massive NG application: 64 encoders, 64 ledrings, 64 RGB LEDs, 24LCD modules organised as over 80 text fields, lots of buttons and normal LEDs. The required NG definition files are also rather massive!

 

Pre-processing the required NG files has certain advantages in this situation:  

• break up the project into multiple files to organise different areas of functionality
• substitute text (e.g values) that occur in many places by a single variable definition, that is easy to change. 
• perform mathematical operations that output text parse-able by NG (e.g "value=4" rather than "value=2+2" ,NG cannot process the latter)
• create concise, function like macros that expand to the long winded entries that NG requires.
• perform the calculations and manipulation required to turn user defined data into sysex message defintions

All of this allows rather major changes in structure to be done easily (e.g. add,remove or change the order of a SRIO PCB in the chain) that would otherwise create a horrible amount of typing! 

 

About M4

I tried the C preprocessor, and although it works for #include files, and #define for text substitution, that's about all that can usefully done with it (e.g there is no way to arbitrarily insert a carriage return in the output, for example).  

So enter a tool designed for the job: M4

It is free, GNU, cross platform, well documented, powerful, and from what I've discovered quite easy to use.

 

Invocation

The following is in a windows batch file, other OS's will be similar.

m4 MySource.m4 > MyOutput.ngc

To set up, I simply located m4.exe, and a required dll in my working directory, in the long term it should be located in the system path. The m4 extension on the input file is optional, any extension can be used on input or output.

 

Include

In the input file any text you want is put, along with any macros you wish to use: first we'll look at a built in one for including the contents of other files:

RESET_HW
include(`EncoderDefs.m4') 
include(`LedRingPatterns.m4') 
include(`Buttons.m4')
 

I this example the contents of 3 files is inserted into the output file along with any other text (i.e preceded with RESET_HW in this example fragment)

 

Define

To substitute one value for any other use define:

define(`OFFSET',`13')
define(`SomeDefaults',`chn=1 type=CC fwd_to_lcd=1 range=0:127 offset=0 ports=1000100000001000 led_matrix_pattern=3')

EVENT_ENC    id=1  fwd_id=LED_MATRIX:eval(OFFSET+1) SomeDefaults
EVENT_ENC    id=2  fwd_id=LED_MATRIX:eval(OFFSET+2) SomeDefaults
EVENT_ENC    id=3  fwd_id=LED_MATRIX:eval(OFFSET+3) SomeDefaults

 

This expands to:

EVENT_ENC id=1 fwd_id=LED_MATRIX:14 chn=1 type=CC fwd_to_lcd=1 range=0:127 offset=0 ports=1000100000001000 led_matrix_pattern=3
EVENT_ENC id=2 fwd_id=LED_MATRIX:15 chn=1 type=CC fwd_to_lcd=1 range=0:127 offset=0 ports=1000100000001000 led_matrix_pattern=3
EVENT_ENC id=3 fwd_id=LED_MATRIX:16 chn=1 type=CC fwd_to_lcd=1 range=0:127 offset=0 ports=1000100000001000 led_matrix_pattern=3

note the use of eval() to calculate a value.

 

There is much, much, more that can be done, but this much should provide some idea, and starting point.

If anyone is interested please post to this thread. 

 

[TK.]

I'm surprised that M4 still exists - I used it ca. 10 years ago for a selfwritten assembler (@work) to get the possibility for writing macros without enhancing the parser.

I agree that it's useful.

On the other hand: if you already go for special languages, why not writing a code generator in a high level language?

This is industrial standard since several years.

 

Here a simple example: http://svnmios.midibox.org/filedetails.php?repname=svn.mios32&path=%2Ftrunk%2Fapps%2Fcontrollers%2Fmidibox_ng_v1%2Fcfg%2Ftests%2Fmax72xx.ngc

(the used perl script is embedded into comments)

 

Best Regards, Thorsten.

[Duggle]

On the other hand: if you already go for special languages, why not writing a code generator in a high level language?

I'd like to see a lot more examples of this in the current context. It's an interesting subject.

I like the way the M4 code is embedded in the unmodified source text, so that if you only use a few macros, for example, the source is uncluttered by printf() everywhere.

I have an open mind.

[TK.]

In perl you could also work without printf

 

E.g. if you would like to add some freetext, just write:

  print <<HERE
# this section
# will be print
# out directly
# e.g. you could write some
EVENT_FOO ...
EVENT_BAR ...
# until here
HERE;

 

Inside such a section you could even insert variables (starting with $<name>)

 

You wouldn't be able to do calculations inside such a section, but you could write:

for($i=0; $i<16; ++$i) {
   my $id = $i + 1;
   my $key = $i + 36;
   print <<HERE
EVENT_FOO id=$id type=NoteOn chn=1 key=$key
HERE;
}

 

Best Regards, Thorsten.

[Duggle]

That's cool.

It's good to know that if I find that M4 stops me doing something I want/need, I can switch to perl.

At the moment they seem roughly equivalent, but I'd bet perl is more powerful/flexible as a language. 

[TK.]

Yes, especially because it would allow you to stream the output into different files.

 

This would solve such a topic: 

 

Means: a single script could write into a .NGC, .NGL and even into a .NGR file.

 

Best Regards, Thorsten.

[Duggle]

This talk of perl is a few days too late! :D I spent all this time exploring the C pre-processor, then discovering M4, and now working on a lovely set macros to handle things really nicely.

(Hey, learning a new language is good for the brain!)

Like I said, translating what I've done to perl should be straight forward when the time comes (maybe soon).

I'm building my ngc files, and ngl fragments with the following *.bat

REM process macro expansion
m4 NGsource.m4 > output.txt

REM new file from lines containing LABEL
sed -n '/LABEL/p' output.txt > labels.ngl

REM new file with lines NOT containing LABEL
sed -n '/LABEL/!p' output.txt > config.ngc

 

Using the standard "stream editor" sed to split off LABEL content for the ngl file.

 

The workflow is to have config.ngc in a window of Notepad++ silently update (after running the above), then CNTRL-A,CNTRL-C (select all, copy) then paste in the MIOS Studio File browser window with DEFAULT.NGC open and CNTRL-A, CNTRL-V (select all, paste) then "save" button. Voila, the changed NCG is running.

 

With the DEFAULT.NGL, I similarly paste the LABEL content from above, as required. The conditional statements in NGL don't change as much, so atm they're manual.

 

[TK.]

Actually I intended to propose the usage of "sed" for this simple filtering job first, but then thought it would be better to avoid overloading you with so many different tools for such similar purposes ;-)

 

Best Regards, Thorsten.