My sddc-skeleton Enhancement Quest... autotools skeleton and libcmios

[Narwhal]

Ok, that seems to have done it!  I hope that it can be worked out for this setup to be more generally used, it's certainly a lot cleaner and easier to understand (for me at least) than having all of these files repeated in every project.

Thanks Didier.

[ptitjes]

Ok, that seems to have done it! 

Cool! Do no hesitate to ask me questions about the configure.in and Makefile.am files of the skeleton, as I still not have finished the documentation of its anatomy. You can also crosscheck the files from the skeleton and the ones from libcan, to better understand. Also here are the unique documentations i used to make that work :

automake manual

autoconf manual

I'll add those links in the Wiki page too!

I hope that it can be worked out for this setup to be more generally used, it's certainly a lot cleaner and easier to understand (for me at least) than having all of these files repeated in every project.

Yeah! I do hope people will come and test the beast. :)

I'm happy that you feel the same as me. I was completely stunned the first time i've seen the current skeleton, trying to understand what files were there for what, and why there were so many...

Thanks Didier.

Thanks to you for your patience.

Cheers, Didier.

PS: I won't be at home for one week to go and have Christmas festivities with my best friend. Do not know if i'll have an Internet access so...

[TK.]

I would propose to compare the number of files and complexity again once a realworld project has been setuped and tested ;-)

Is it really required to create so many derivative specific links to the wrapper in the bin/ directory? Why not using a generic keyword for the PIC18F family like "pic16" instead of "pic18fxxxx"?

What I'm missing is the possibility to change the stack sizes for main task and interrupts (as the wrapper is located in a prebuild library, it isn't possible to change it via define anymore). Solution: the stack pointers have to be defined via external constants

How is it possible to adapt the linker file for application specific locations? How should this be handled for different derivatives? See also http://www.midibox.org/dokuwiki/using_pic18f4620#linker_script

How is the version control managed? In long term it can happen, that users release an application which is only qualified for a specific MIOS version, or a specific library element, cmios wrapper or SDCC version.

The last one is the most important, as the cmios wrapper implementation heavily depends on the way how SDCC passes parameters.

Best Regards, Thorsten.

[ptitjes]

I would propose to compare the number of files and complexity again once a realworld project has been setuped and tested ;-)

As you can see there: http://www.midibox.org/dokuwiki/howto:dev:autotools-skeleton#getting_started

4 files added to the application itself (main.c and README)

Is it really required to create so many derivative specific links to the wrapper in the bin/ directory? Why not using a generic keyword for the PIC18F family like "pic16" instead of "pic18fxxxx"?

I chose the way it is done everywhere else! Every time you install a cross-compiling toolchain you get bin tools. But is it a problem ? It is transparent to the user...

What I'm missing is the possibility to change the stack sizes for main task and interrupts (as the wrapper is located in a prebuild library, it isn't possible to change it via define anymore). Solution: the stack pointers have to be defined via external constants

Good point! ;) I've seen this one while scrubing for in the wrapper... This one is a real problem and I've only thought a little about this. But maybe we can find a trick so that the wrapper find those pointers elsewhere!

How is it possible to adapt the linker file for application specific locations? How should this be handled for different derivatives? See also http://www.midibox.org/dokuwiki/using_pic18f4620#linker_script

Standard linker scripts (adapted for MIOS) are provided by default for the host platform you compile libcmios for. They are transparently selected by the toolchain-wrapper.

But you can still add a

LDFLAGS += -s my-own-script.lkr

in your Makefile.am file.

How is the version control managed? In long term it can happen, that users release an application which is only qualified for a specific MIOS version, or a specific library element, cmios wrapper or SDCC version. The last one is the most important, as the cmios wrapper implementation heavily depends on the way how SDCC passes parameters.

Tests like AC_CHECK_HEADERS(...) used in the configure.in are programmed in the M4 macro language. I already use this for the MIOS-specific tests i included in mios-pic16-toolchain. There are plenty of standard tests available in Autoconf. And you can easily progam new tests to check for new things. So:

• Specific MIOS version: should rely on libcmios version
• Specific library element: AC_CHECK_HEADERS(...) There is an example of this in skeleton. There are lots of other AC_CHECK_ for libs, headers... see Autoconf documentation.
• cmios wrapper or SDCC version: I can add another specific test M4 macro in mios-pic16-toolchain to support this. That's no problem at all!

What is interesting with Autoconf tests is that they automatically provide C preprocessor symbols (in generated config.h) so you can even have different way of doing things in your code depending on the availability of a library, header or anything you test in configure.in. I use this feature to have __pic18f4685__ or __mios__ defines in the skeleton. Example in my libcan/configure.in:

AC_ARG_ENABLE([debug], [  --enable-debug  (requires libdebug)], [
        AC_CHECK_HEADERS([debug.h], [
                AC_DEFINE([DEBUG], [], [Whether debug is enabled or not])
        ], [
                AC_MSG_FAILURE([Can't with debug support : libdebug's debug.h not available])
        ])
])

Explanation : If --enable-debug is set in configure then i test for the availability of debug.h. If debug.h is available then defines a preprocessor DEBUG define else fail the configure with a nice message. In the C code i have at the begining

#ifdef DEBUG
#       ifndef HAVE_DEBUG_H
#               error "Can't have debug enabled without libdebug"
#       endif
#       include <debug.h>
#endif

and #ifdef DEBUG #endif sections around each of my debug calls...

Easy and clean!

Best regards, Didier.

[audiocommander]

hm... did I understand this right, that I first have to install the pic16 toolchain?

I tried this on OSX 10.4.11. This is what I got right now:

ezri:/downloads/mios-autotools-skeleton-1.9f-r1/mios-pic16-toolchain-1.9f-r1 zeichensprecher$ ./configure --target=pic18f4685-mios --prefix=/usr

checking build system type... powerpc-apple-darwin8.11.0

checking host system type... powerpc-apple-darwin8.11.0

checking target system type... pic18f4685-microchip-mios

checking for a BSD-compatible install... /usr/bin/install -c

checking whether build environment is sane... yes

./configure: eval: line 1: unexpected EOF while looking for matching `''

./configure: eval: line 2: syntax error: unexpected end of file

configure: WARNING: `missing' script is too old or missing

checking for a thread-safe mkdir -p... ./install-sh -c -d

checking for gawk... no

checking for mawk... no

checking for nawk... no

checking for awk... awk

checking whether make sets $(MAKE)... yes

checking whether ln -s works... yes

checking for rm... rm -f

configure: creating ./config.status

./config.status: line 780: unexpected EOF while looking for matching ``'

./config.status: line 801: syntax error: unexpected end of file

Am I missing something?

ps:

no stress from my side, don't want to distract your attention from more important things, Didier ;-)

if I succeed, I'll complete the OSX part in the wiki :)

[ptitjes]

Maybe try:

autoreconf --install

before configure and make.

Also, I upgraded the documentation which now contains lots of informations (I say this in case you would not have seen this page)

http://www.midibox.org/dokuwiki/howto:dev:autotools-skeleton

I can't help you too much now as I go for one week in one hour.

Merry Christmas!

Cheers, Didier.

[TK.]

So:

• Specific MIOS version: should rely on libcmios version
  
• Specific library element: AC_CHECK_HEADERS(...) There is an example of this in skeleton. There are lots of other AC_CHECK_ for libs, headers... see Autoconf documentation.
  
• cmios wrapper or SDCC version: I can add another specific test M4 macro in mios-pic16-toolchain to support this. That's no problem at all!
  

What is interesting with Autoconf tests is that they automatically provide C preprocessor symbols (in generated config.h) so you can even have different way of doing things in your code depending on the availability of a library, header or anything you test in configure.in.

I use this feature to have __pic18f4685__ or __mios__ defines in the skeleton.

Example in my libcan/configure.in:

AC_ARG_ENABLE([debug], [  --enable-debug  (requires libdebug)], [
        AC_CHECK_HEADERS([debug.h], [
                AC_DEFINE([DEBUG], [], [Whether debug is enabled or not])
        ], [
                AC_MSG_FAILURE([Can't with debug support : libdebug's debug.h not available])
        ])
])[/code]

Explanation : If --enable-debug is set in configure then i test for the availability of debug.h. If debug.h is available then defines a preprocessor DEBUG define else fail the configure with a nice message.

In the C code i have at the begining
[code]#ifdef DEBUG
#       ifndef HAVE_DEBUG_H
#               error "Can't have debug enabled without libdebug"
#       endif
#       include <debug.h>
#endif

and #ifdef DEBUG #endif sections around each of my debug calls...

Easy and clean!

For programmers like us this is easy to handle after a short learning phase, especially because these are general concepts which are used by many projects today, so it isn't wasted effort to get familiar with such generic approaches.

But IMHO, you still see the whole thing too much from the programmers perspective, and it seems that you expect from every MIOS user (even if he doesn't want to get into this topic, and only wants to change little things to customize the application) to spend some hours or days (maybe also with additional support from the forum) before beeing able to rebuild a .hex file, something which can be done today within a couple of minutes, and without the danger for incompatibility issues.

I will give an hypothetical example:

(1) Programmer A releases a quite useful library element

(2) Programmer B uses this element in his application, his release package only contains his own source code.

(3) Programmer A is doing changes in his library element, but Programmer B doesn't find them useful, as they slowdown his application, therefore stays at the old version.

(4) Programmer A is doing even more changes. Parameter lists in functions are enhanced, so that the lib can even not be compiled anymore with the app of programmer B

(5) after one year, a user has to recompile the application of programmer B. The compiler fails since he only owns the latest lib release

(6) after some postings in the forum we find out, that he is using an incompatible release (programmer B wasn't aware of the funky autoconf checks, therefore the user wasn't warned about this)

(7) programmer B doesn't respond to emails. With help from the community we find an older library version which could be compatible. Unfortunately the application is slowed down, realtime conditions are not met anymore (see 3.)

(8) user is unhappy about his MIDIbox project, since it's so slow and buggy. Nobody can really help him, since only he and unreachable programmer B own exactly the hardware which is required to reconstruct issues.

Something what is working quite well in the unix world meanwhile, cannot be 1:1 adapted to a project like MIDIbox. When somebody releases a complex application, he is the only one who can test it with his dedicated hardware for a long time, until other users build the same. Mostly he stays the only programmer who knows how to maintain the project. and he is the only guy who can qualify library changes with his hardware.

Instead of beeing able to release a snapshot of all sources he was using during the programming period (like in he current approach), he releases only his own part. If he doesn't consider all the required compatibility checks due to lack of knowledge with the toolchain, and if the external parts he was using are not properly released, nobody will be able to reconstruct the .hex file after one or two years, especially when the programmer is not available anymore.

Another issue for which I don't see a solution in your current implementation: how should a common user without programming knowledge recompile different MIDIbox projects which where created at different times (we are speaking about years), and which require different library or SDCC versions? Should he overwrite the /usr/local/mios files each time with older versions whenever he wants to create a new binary, or should he create parallel path structures? At least then we have reached exactly the state as today -> directories with many files... ;-)

Sorry for the long posting before christmas (I will make a break as well) - just take it as inspiration for improvements. :)

Best Regards, Thorsten.

[ptitjes]

Hi TK, hi all!

First i hope you had a good Christmas and i wish you all an Happy New Year, health and happiness... hardware and coding success too :)

TK, please don't take offense at what i write in this post. I read you many times hoping that i understand it wrong. However, i don't find your argument about versionning and modularity relevant. Let me explain why...

(Hope you will read that long post patiently... sorry!)

---

It seems to me that you use the versionning of modules as an argument against modularity itself.

The whole programming industry (yes all! and not only open-source and *nix communities) made programming evolve to an art of feature-separation and good modularity. I don't want to use that fact as an argument, I just want to remind you that such a discussion already occurred all over the place many years ago. There are lots of reasons why this direction has been taken and maybe we should learn from those reasons... (the first surely being maintainability)

In fact, a better modularity makes less version changes as, obviously, orthogonal features evolve independently in different modules. Thus, single changes in independent modules are easier to handle than multiple changes in a single software brick.

As for the interdependence and compatibility between modules and applications. A simple, yet efficient, way of doing things exists : a standard versionning scheme. One exists that has proven its benefits : major.minor.release

• Major version change implying API compatibility change (and obviously binary) and/or addition/removal of sets of new features
  
• Minor version change implying binary compatibility change only and/or addition of little new features
  
• Release number change implying bug fixes only
  

Deprecation, along to this versionning scheme, can be used to make things evolve between version without forcing module users to update their applications at every module change.

Also, modules for orthogonal feature sets makes users able to take benefit of new features and bug fixes more quickly. For instance, if one has a good led/button matrix code and make a library for it, then another don't have to replicate/copy that code in its own application. When the matrix code evolves, then all users can take profit of those changes without updating their code, by just updating the library (provided that this is just a minor version change or that the old API is only deprecated) and recompile their application. Using the standard version scheme, knowing that the library is still compatible is only a matter of comparing the version numbers.

When one releases an application, he writes in the documentation a requirement section where he tells what modules and versions he has used (or announce it on the forum post). Again, knowing that a new library version is still compatible is only a matter of comparing the version numbers.

When a new major version is set up (with new features), old versions of a library should be provided for download. Also the last old major version could be maintained temporarily (what is called branching) with bug fixes, in order to ease transition and not force users to update immediately.

---

Making a feature evolve in a big and fuzzy application is quite complicated because it is harder to know how changing a thing in one place won't interact with something in another place.

Making a feature evolve in one library is thought based on API evolution. You know what you can do and what you cannot do in order to no break compatibility. You know what a particular change will imply for users. And you know that you can change the internals of the library without breaking API compatibility.

Also, having defined and cleanly separated APIs for different feature set is very very convenient for programming users. It is easier for them to have a big picture of the whole architecture and/or of their own application! One can better identify the interaction between software and hardware without delving into the code.

Finally, having different modules for different feature set enables users to choose appropriately what features they need and update their versions independently. They don't bloat their head with informations or features not useful to them.

I understand this may be a little harder to achieve for module providers but this is not the job of the non-programming users!

---

The fact that a non-programming user should be able to adapt an application to its own hardware is an orthogonal problem. I agree this should be simple to execute. But this should not make complex thing hard to do when they could also be simple. So let resolve this use case by making it simpler even. IMHO, non-programming user should not have to recompile an application to adapt it to their hardware (provided the feature field is the same). POINT.

I can give you a possible track to explore to solve this in a clean/easy way. It seems to me that configuration tables should be provided in a second upload. There is no need for a C or even asm compiler to generate those tables... a script and appropriate tooling integration (MIOS Studio...) could do the thing. Then one could provide binaries for its application (in addition to the sources if he wants to release them).

---

Again, let give me an example similar to yours to demonstrate how this can be evil to manipulate an application with the current approach :

(1) Programmer A writes an application based on the actual MIOS C wrapper.

(2) Programmer B likes parts of what this application does and replicates the application code to integrate those parts to its own application.

(3) Programmer B also makes changes to that code in order it fits the way its application works.

(4) Meanwhile, Programmer A fixes many bugs in his application and maybe is doing even more changes. (Like you said : parameter lists in functions are enhanced...)

(5) Programmer B tries to back-port bug fixes in its own application, but the code is not anymore compatible. When he copies/pastes the code, he even introduces new bugs (and maybe compile errors)

(6) Programmer B explains on the forum how his application doesn't work since he back-ported the code of A, but programmer A can't identify where the problems comes from as the code of B is now very different.

(7) Programmer B is unhappy about his MIDIbox project since it can't even work anymore...

That example is not so hypothetical and I'm sure I could find many forum posts where that kind of problems already occurred.

We can finally conclude from that experience that back-porting sucks so that modules is the way to go.

Best regards, Didier.

[bugfight]

don't forget this is an embedded system, with severe resource limitations.

i'm a big fan of OOP, but not for PIC...

[TK.]

Hi Didier,

I don't take this as offence, and hope that you don't take it personally when I'm saying, that I still have the impression that you are trying to apply methods evolved in computer world to the embedded world, regardless of the facts that PIC microcontrollers have very limited resources which don't allow to use all those nice programming techniques like soft-configuration, sophisticated APIs, complete modularity, etc...

Of course, in small embedded applications your suggested approach will work in it's whole beauty, but once resources should be shared between complex applications created by different people, it won't work anymore, or you will loose so much performance that general programming rules have to be violated again and again (and exceptions make it even more worse).

I want to highlight: I'm speaking about projects for 8bit controllers. It's almost the same for 16bit controllers, it has changed for 32bit controllers due to enhanced instruction sets which support C and C++ programming, and the huge amount of memory.

I for myself use a totally different programming style on 32bit controllers, which goes into the direction you are suggesting here. Based on my experiences you cannot reach the same performance on 8/16bit controllers with a full modular approach (where sources are completely statically released in libraries)

In addition, your approach can only work when program sources are released in a version management system. @Work I'm using Clearcase to share sources with other team members, in open source projects CVS is prefered. Such a system is at least required to make changes smooth, e.g. giving somebody the possibility to incorporate library changes made by other people, and to label/tag a working version. It also gives the possibility for users, to get a released version view with "known working" modules which have been tested by the maintainer (what I called "qualified" release before)

I think that it isn't required to give a comment on all your statements. Many of them are true and I know them from other parties as well, I even apply them by myself in other projects. Some others are realstic for MIOS projects, they just don't exist because nobody have implemented them yet. Some others are over-engineered, or just go into the wrong direction, which you will probably notice two or three years later once you made your own experiences with 8bit programming, and especially with the people who are programming!

I will give you three examples and I would like to know how you would handle them:

1) http://www.ucapps.de/mios/midibox_lc_v2_0_alpha1.zip: how would you implement the "hard-options" in main.h in a configuration layer without consuming more memory than available on a PIC18F452? Note that the same source code is used to create a .hex for MIDIbox LC and MIDIbox SEQ - two totally different hardware variants, therefore so many #define statements are used.

2) http://www.ucapps.de/mios/midibox64e_v2_2.zip: similar question: how would you handle all the options in setup_*.asm for all the different variations without loosing too much memory?

And how much programming effort do you estimate for an additional configuration layer (and tools which handle the configuration) compared by using simple #define statements? Who should maintain the configuration tool?

How should the configuration layer be structured, so that it's flexible enough for future expansions? Where should configuration data be stored? In RAM, internal Flash or external EEPROM?

(Sidenote: we already had this situation in PIC16F projects - a single firmware and a configuration tool called "vmidibox64" - using #define's was the next step, learned from the limitations and additional efforts of this approach)

3) http://www.ucapps.de/mios/midi_router_v1_1.zip: how would the configuration layer for router.c look like without loosing flexibility.

And how would you make a library element of iic_midi.asm without static pin assignments PORT_RI_N_SLAVE_* (again: without loosing too much performance)

Best Regards, Thorsten.

[ptitjes]

don't forget this is an embedded system, with severe resource limitations.

i'm a big fan of OOP, but not for PIC...

Hi BugFight!

I know what embedded systems are. And who speaks about OOP here ? Modularity is not OOP. So please read the posts completely before trolling...

TK:

The suggestion about a second upload for configuration was just a flying idea. Not thought about this too much but I think I'll explore that in the following months...

I know 8bit micro-controllers does not enable full-modularity and state of the art programming. However when you have already C functions that are already defined in a header and independent of any other part of a program please tell me what prevents you to put that in a separate library...

I won't go back to the actual skeleton for the following reasons :

- I have many applications and prefer to update one library (libcmios) than many applications when MIOS will be updated or new processor will be available.

- Adding a sub-folder for sources took 4 hours of work with its proprietary makefile generator where i had one day of work to get the autotools toolchain/skeleton to work and now i need 15 seconds to add a sub-source folder.

- Making libraries is impossible if you don't want to have a one day rework of the proprietary makefile generator where the autotools toolchain/skeleton provides that as is...

I don't want to fight with the MBHP community but help it go forward. So please accept my point of view.

I think we should stop that discussion as this goes nowhere. I was trying to push some things i think are good for the MBHP. Anyway, i will continue to release/maintain the toolchain/skeleton and my libraries, if someone wants to use that! I still hope you'll change your mind and maybe, one day, try the stuff I released to really see how it works and how simple it is!

Best regards, Didier.

[TK.]

Hi Didier,

I know 8bit micro-controllers does not enable full-modularity and state of the art programming. However when you have already C functions that are already defined in a header and independent of any other part of a program please tell me what prevents you to put that in a separate library...

Nobody asked for library releases yet...

Some of the modules could be put into a library, some others not since they are using constant definitions to customize the code. I don't like different approaches for different usecases. In addition, I would prefer that all source files which are required to rebuild an application are part of the release package.

Therfore a good starting point would be a wiki page which collects all code modules, that are already available (there are a lot, but most of them are not so nicely documented like your CAN library).

- I have many applications and prefer to update one library (libcmios) than many applications when MIOS will be updated or new processor will be available.

MIOS: was always downward compatible in the past, you can still run applications which have been programmed and compiled for earlier versions with the latest MIOS version

Processor: within the PIC18F family, there are not so many processors which are qualified for MIOS due to available peripheral functions and packages. All of these processors are compatible to PIC18F452, only the available memory differs. Ok, there is one exception: the PIC18F4685 (and similar CAN derivatives) has a memory window between 0x60..0x7f which is not directly accesible.

But if somebody wants to use a PIC18F452 application on a PIC18F4685, he only has to change a single line in the default linker script. This is a question of how good such a change is documented.

Such a manual change would even be required in your approach, if a local linker file has been used in the project (e.g. for larger arrays)

Or did I overlook something?

Adding a sub-folder for sources took 4 hours of work with its proprietary makefile generator where i had one day of work to get the autotools toolchain/skeleton to work and now i need 15 seconds to add a sub-source folder.

Why haven't you just asked me for an extension in the makefile generator instead of setting up a new flow which will cause additional efforts at other sides?

I will add this possibility to the next skeleton release.

- Making libraries is impossible if you don't want to have a one day rework of the proprietary makefile generator where the autotools toolchain/skeleton provides that as is...

Here again: why haven't you just asked me to add such a feature? It doesn't take much time for me to add such a function, especially if I find it useful as well - it will be primitive, but I can ensure that it will work sufficient enough for Unix and DOS.

I don't want to fight with the MBHP community but help it go forward. So please accept my point of view.

Please also accept that I'm trying to keep things simple enough. There are flaws in your concept, and they can only be solved by adding even more complexity (tools and configuration options). Partly these are issues which went into my own decitions in the past, and which lead to the setup as it is.

I fear that your proposals complicate the whole flow too much for a project, which is mainly intended for people who like to make music, and who mostly don't have the technical background like IT guys.

I think we should stop that discussion as this goes nowhere. I was trying to push some things i think are good for the MBHP. Anyway, i will continue to release/maintain the toolchain/skeleton and my libraries, if someone wants to use that! I still hope you'll change your mind and maybe, one day, try the stuff I released to really see how it works and how simple it is!

I'm disappointed that your contributions will only be released for your own approach, but I guess that this shouldn't prevent others to try the same, time will show if it works successfull.

What I take from this debate:

- there is the need to define (sub)directories in the mkmk.pl script

- it should be possible to generate a Makefile for a library

- consider to provide the wrapper as a precompiled library (*)

- there is still no page which collects all code module contributions

(*) here I could profit from your work: did you already found a way to define the stack pointers outside the library?

Concerning the two tables in mios_tables.inc: I noticed that you defined them in the main.c file - I would prefer following approach: put a mios_mproc_event_table.o and mios_enc_pin_table.o file into the library. These objects should contain the default table content. Then add "extern" references to these tables into cmios.h

This opens following possibilities: the tables will be preloaded correctly with default entries, taken from the library, whenever there is no .o file with *the same name* in the local scope during linking. Accordingly, the programmer is able customize the tables within his project by copying the .asm file(s) from the library source directory into his own project, doing the required modifications and compiling them locally.

Best Regards, Thorsten.

[bugfight]

... So please read the posts completely before trolling...

i strongly suggest that you reconsider that statement..

[ptitjes]

Hi TK,

Therfore a good starting point would be a wiki page which collects all code modules, that are already available (there are a lot, but most of them are not so nicely documented like your CAN library).

I can only agree with this!! This first thing I can point out is the led/button matrix thiing that really would need to be in a library. (this is already on my todo list as I need that in two of my applications) But there are lots of other code spread into many of the applications that could be factored out for the benefit of everyone... +1 for the wiki page.

MIOS: was always downward compatible in the past, you can still run applications which have been programmed and compiled for earlier versions with the latest MIOS version

Sorry, I was talking for the new stuff in new MIOS versions. When there is a new thing available in MIOS, i don't want to have to replicate the new wrapper code everywhere to benefit of this new thing. A simple such example that I have been in the place to watch is the addition of the DETENDED3 constant.

Processor: within the PIC18F family, there are not so many processors which are qualified for MIOS due to available peripheral functions and packages. All of these processors are compatible to PIC18F452, only the available memory differs. Ok, there is one exception: the PIC18F4685 (and similar CAN derivatives) has a memory window between 0x60..0x7f which is not directly accesible.

But if somebody wants to use a PIC18F452 application on a PIC18F4685, he only has to change a single line in the default linker script. This is a question of how good such a change is documented.

Such a manual change would even be required in your approach, if a local linker file has been used in the project (e.g. for larger arrays)

Or did I overlook something?

Sorry, again, i was not that clear. I was meaning complete new processor family. But here I was really anticipating. Please just forget that if you find thinking about this is too soon.

As for the linker scripts, if you need a project-specific script you have only on line to add in the Makefile.am. But please remember that memory windows larger than what is addressable by the PIC is greatly discourage by both the PIC and also sdcc documentations.

Also, in my approach, if you do not need a project-specific linker script, a processor-specific linker script is already selected transparently without any user intervention at all.

Why haven't you just asked me for an extension in the makefile generator instead of setting up a new flow which will cause additional efforts at other sides?

I will add this possibility to the next skeleton release.

Because, as I've read many times on the forum, you already have lots of topics to work on, and that you are the sole one that can work on these. I do prefer that you work on all those things that will bring us additional advantages to use MBHP.

Understand my point of view : I am a newbie in electronics. In more than six months now, I started reading The Art of Electronics. I also read quite all, I think, the documents I could find on the Wiki and ucapps.de, delved into the code of quite all the applications (not started reading th SID one yet though). I am, day by day, understanding more and more of your work. I now have a quite clear understanding of all the MBHP architecture, and the interactions between hardware and software.

However, I still miss some features and this is why I consider the box I'm building will only be a prototype. So when I can do some things (like software, which is what I'm really good ine for now), I do it to let you work on other things I could not do for you.

Please also accept that I'm trying to keep things simple enough. There are flaws in your concept, and they can only be solved by adding even more complexity (tools and configuration options). Partly these are issues which went into my own decitions in the past, and which lead to the setup as it is.

I fear that your proposals complicate the whole flow too much for a project, which is mainly intended for people who like to make music, and who mostly don't have the technical background like IT guys.

However, those really fit my needs. Please see below...

And also, I don't understand why you want to reinvent the wheel. There exists software that do what your makefile generator does and more. So why don't use it. I'm tired of people rewriting things already existing, because they can't reach the consensus that brought those things to exist. (Please don't take this as an offense, as you have reasons that we are discussing in the past posts)

I'm disappointed that your contributions will only be released for your own approach, but I guess that this shouldn't prevent others to try the same, time will show if it works successfull.

What I take from this debate:

- there is the need to define (sub)directories in the mkmk.pl script

- it should be possible to generate a Makefile for a library

- consider to provide the wrapper as a precompiled library (*)

- there is still no page which collects all code module contributions

But you forget all the other features I really need : (you can take that as a feature request...  if you really want to code that again)

- Checks for the host platform if the application/library can be build for it (eg. my libcan can only be build for 4685)

- Checks for presence of some headers for the building platform and automatic preprocessor symbols to have code variants

- Checks for presence of specific C compiler features

- Possibility to have features enabled/disabled at configure time and automatic preprocessor symbols to have the corresponding code variants (eg. my libcan can be build with or without debug support, and i will add more features in the future like disabling of inline assembler code optimized variants, ...)

- Automatic distribution releases building (dist, dist-zip, ...) that really ease the making of a source distribution archive

- Automatic installation of documentation/licenses in a standard location and other data along with the install of a library

All of these comes by default with AutoTools and the toolchain I released.

And the ones you pointed out :

- Checks for libraries' and MIOS versions

- ...

Also, how will you manage the location of the libraries' headers to build applications that use them ? Will you have to configure each and every application with the location of the headers of each of the libraries it uses ? This can be done in an automated way which is what 'make install' are for. Also if you rewrite some code for a 'make install' in your makefile generator, you will have a standard path where to put libraries and their headers, coming back to problem of which platform those libraries are for people that build applications for different PIC micro-controllers.

Please, make the sum of all those things we just enumerated above and estimate the amount of dev/test/... you still have to do to reach the quality of the AutoTools.

Also, please consider that using standard tools is a way to make the MIDIbox project more professional and accessible to people already involved in other open-source projects.

(*) here I could profit from your work: did you already found a way to define the stack pointers outside the library?

Concerning the two tables in mios_tables.inc: I noticed that you defined them in the main.c file - I would prefer following approach: put a mios_mproc_event_table.o and mios_enc_pin_table.o file into the library. These objects should contain the default table content. Then add "extern" references to these tables into cmios.h

This opens following possibilities: the tables will be preloaded correctly with default entries, taken from the library, whenever there is no .o file with *the same name* in the local scope during linking. Accordingly, the programmer is able customize the tables within his project by copying the .asm file(s) from the library source directory into his own project, doing the required modifications and compiling them locally.

I have to test what you suggest here. If it works that is a good thing.

Edit: I'm currently designing my PCBs, so I can't work on this now. I think I have a month of work to get some correct PCB layouts (I have 16 PCBs to layout... !!$##** :)) However, the toolchain/skeleton provide quite all of what I need now, so, after the PCB layouts are ready, I will then polish it while completing/finishing my libraries/applications.

Hoping that all these issues will see their resolution in 2008 ;)

Best regards, Didier.

[TK.]

Hi Didier,

Sorry, I was talking for the new stuff in new MIOS versions. When there is a new thing available in MIOS, i don't want to have to replicate the new wrapper code everywhere to benefit of this new thing. A simple such example that I have been in the place to watch is the addition of the DETENDED3 constant.

Yes, thats the drawback when complete source packages are released. On the other hand I can give users simple instructions how to update the MIOS wrapper. They only need to know, how to copy files from one into another directory.

E.g., each MIOS release contains a migration/ directory with files which can be copied over the original files of a release package. By concept they are always downward compatible.

In your approach, the user has to install a new MIOS library package. Thats similar, it has some other advantages, but also disadvantages (risk that some additional files or code libraries are released, which are not compatible with the application - bad if the user faces such issues again before the maintainer has tested a new library release with his hardware), but I think that we don't need to discuss this again - it would really be endless ;-)

Sorry, again, i was not that clear. I was meaning complete new processor family. But here I was really anticipating. Please just forget that if you find thinking about this is too soon.

The next processor family used in MBHP is probably a 32bit microcontroller, either ARM or TriCore based. Regardless of which device is finally used, the code base has to be completely overworked. Re-use of existing files won't be possible - the trouble starts already with SDCC incompatibilities to gcc (e.g. structures, word alignments), assembly code, but also general conceptional topics. E.g., for a future OS I would like to include sophisticated interrupt handling, and multitasking capabilities (or I could be on a free OS like FREERTOS, or maybe Linux?).

The programming style will be totally different (and hopefully easier)

Accordingly it would be required to setup a new software platform - with a totally different structure. It wouldn't make much sense anymore to provide MIOS as a single kernel, also the bootloader wouldn't exist anymore (MIDI is too slow to transfer huge memory dumps). The whole firmware would be uploaded on a single run, probably via JTAG.

The point is: for such a platform change I would prefer the usage of an existing toolchain, maybe even on an existing IDE for people, who want to spend some money for powerful debugging tools (and you really need such tools once you are starting to work with foreign sources). It could be structured totally different from yours. E.g., I evaluated HiTOP in the past, it generates Unix compatible Makefiles and uses Cygwin in background under Windows which will please you (as it means: the source code origanisation and build process is Unix compatible)

But this just as an example - it could also look totally different in some years (once I'm doing the platform change...) depending on what is available at this time. If a powerful non-commercial toolchain is available until then, I would prefer such a solution instead of a (low-cost) commercial environment of course!

And also, I don't understand why you want to reinvent the wheel. There exists software that do what your makefile generator does and more. So why don't use it. I'm tired of people rewriting things already existing, because they can't reach the consensus that brought those things to exist. (Please don't take this as an offense, as you have reasons that we are discussing in the past posts)

So, you know the requirements: generating a simple DOS batch file - this is not possible with autoconf, but it's also obsolete as you can completely work with Unix tools once Cygwin has been installed.

Your argument is, that it's easy for Windows people to install the additionally required tools.

My argument is, that it will increase the complexity too much, including the installation of more files on the computer (can take a long time if somebody only has only a low-bandwith connection to internet (modem)), and learning how to use the bash shell instead of doubleclicking a .bat file

But before discussing in circles again: some Windows guys - not only programmers, but especially "common users" - should try out your approach and report their experiences.

But you forget all the other features I really need : (you can take that as a feature request...  if you really want to code that again)

Not really ;-)

- Checks for the host platform if the application/library can be build for it (eg. my libcan can only be build for 4685)

Nice gimmick, but nobody would try to build a dedicated PIC18F4685 application for PIC18F452 - and if he doing this, he will notice very quickly that some SFRs are not available during build time.

- Checks for presence of some headers for the building platform and automatic preprocessor symbols to have code variants

The presence of headers doesn't need to be checked if all sources are part of a release package (and the maintainer ensured, that a rebuilt is possible), and preprocessor symbols can currently be easily included from the MAKEFILE.SPEC (-D...)

- Checks for presence of specific C compiler features

Some C compiler features were unstable in the past (see bug tracking), it's dangerous to allow such a flexibility - you will notice this once you are debugging code anomalies like I did in the past.

- Possibility to have features enabled/disabled at configure time and automatic preprocessor symbols to have the corresponding code variants (eg. my libcan can be build with or without debug support, and i will add more features in the future like disabling of inline assembler code optimized variants, ...)

same can be defined in MAKEFILE.SPEC

- Automatic distribution releases building (dist, dist-zip, ...) that really ease the making of a source distribution archive

The release package which contains all sources is per concept ready for distribution (ok, "make clean" before zipping it makes sense)

- Automatic installation of documentation/licenses in a standard location and other data along with the install of a library

...so far somebody writes documentation...

Also, please consider that using standard tools is a way to make the MIDIbox project more professional and accessible to people already involved in other open-source projects.

Moving the platform into a direction which makes it only understandable for an elitary circle of Unix geeks?

Maybe I see it too much black-white, I would like to hear the oppinions of other programmers and especially users - if they still follow this thread ;-)

Best Regards, Thorsten.

P.S.: I did the changes in the makefile generator this morning, and going to do some compatibility checks with SDCC 2.7.0, which I was planning for a long time... seems that the imperfections of 2.6.0 have been fixed! :)

[ptitjes]

Re TK,

The next processor family used in MBHP is probably a 32bit microcontroller, either ARM or TriCore based. Regardless of which device is finally used, the code base has to be completely overworked. Re-use of existing files won't be possible - the trouble starts already with SDCC incompatibilities to gcc (e.g. structures, word alignments), assembly code, but also general conceptional topics. E.g., for a future OS I would like to include sophisticated interrupt handling, and multitasking capabilities (or I could be on a free OS like FREERTOS, or maybe Linux?).

The programming style will be totally different (and hopefully easier)

Accordingly it would be required to setup a new software platform - with a totally different structure. It wouldn't make much sense anymore to provide MIOS as a single kernel, also the bootloader wouldn't exist anymore (MIDI is too slow to transfer huge memory dumps). The whole firmware would be uploaded on a single run, probably via JTAG.

That would be great if you could take some time to write in greater detail a wiki page with the different directions you are exploring, and your actual state of mind for your future architecture.

The point is: for such a platform change I would prefer the usage of an existing toolchain, maybe even on an existing IDE for people, who want to spend some money for powerful debugging tools (and you really need such tools once you are starting to work with foreign sources). It could be structured totally different from yours. E.g., I evaluated HiTOP in the past, it generates Unix compatible Makefiles and uses Cygwin in background under Windows which will please you (as it means: the source code origanisation and build process is Unix compatible)

But this just as an example - it could also look totally different in some years (once I'm doing the platform change...) depending on what is available at this time. If a powerful non-commercial toolchain is available until then, I would prefer such a solution instead of a (low-cost) commercial environment of course!

Do not know any of those... I currently concentrate my background research around FPGAs.

But before discussing in circles again: some Windows guys - not only programmers, but especially "common users" - should try out your approach and report their experiences.

People! People! Please could you help by testing ?? :)

Maybe could you contact directly people that you know could compose a nice sample of users with different methods ??

Nice gimmick, but nobody would try to build a dedicated PIC18F4685 application for PIC18F452 - and if he doing this, he will notice very quickly that some SFRs are not available during build time.

I do prefer a nice explanation message at configure time than an ugly and obscure C compiler message that would only add some more support requests...

The presence of headers doesn't need to be checked if all sources are part of a release package (and the maintainer ensured, that a rebuilt is possible), and preprocessor symbols can currently be easily included from the MAKEFILE.SPEC (-D...)

What do you call a release package ?? Are you saying that library headers should be copied in every application. If yes you are coming back to the same problems as it is actually...

You have to understand that those preprocessor symbols are defined or not whether a header file is present. This is way far more powerful. And library headers are shared amongst all applications.

Some C compiler features were unstable in the past (see bug tracking), it's dangerous to allow such a flexibility - you will notice this once you are debugging code anomalies like I did in the past.

But it is possible, that some software need them. Having a check for those is better than testing for a particular sdcc version. And again you'll have a pretty explanation error message at configure time.

same can be defined in MAKEFILE.SPEC

I do prefer command-line configuration than having to edit a file from the distribution. Also, one has to know what preprocessor symbol it has to define to have different build features. You only have to do "./configure --help=short" to have a clear and documented view of what build features a particular package has.

The release package which contains all sources is per concept ready for distribution (ok, "make clean" before zipping it makes sense)

Yeah, renaming the directory with correct version number, making a clean, and then zipping...

"make dist" is way far simple for anybody.

...so far somebody writes documentation...

Oh! What a dishonesty... :) We are all, you included, complaining about the lack of documentation on some topics and in general that documentation is fuzzily organized on the Wiki.

Providing a way to have corresponding documentation along to the code is a good way make things more organized. So you should support that if you want more documentation.

Moving the platform into a direction which makes it only understandable for an elitary circle of Unix geeks?

Maybe I see it too much black-white, I would like to hear the oppinions of other programmers and especially users - if they still follow this thread ;-)

I don't think that this is so unix-oriented. Also, i still hope some people with a long background of your project could try the proposed a skeleton and give feedback.

P.S.: I did the changes in the makefile generator this morning, and going to do some compatibility checks with SDCC 2.7.0, which I was planning for a long time... seems that the imperfections of 2.6.0 have been fixed! :)

Yeah i told you that in one of the first posts in this thread. However, warn that (i don't have the commit mailing-list threads at hands) for support of the 4685 you need svn version of sdcc and latest version of gputils.

Best regards, Didier.

[TK.]

Concerning the usability evaluation: I would propose a two-step approach: first ask two or three programmers of User Projects to migrate their source code, thereafter ask common users (in the MIDIbox news section) to rebuild the applications (it isn't required to own the appr. hardware)

Goal of the evaluation should be to find out, how much support people need to install and troubleshoot the toolchain, and how long it does take to get the setup ready for rebuilding an application.

Please highlight in the announcements, that we are not searching for hackers who are already familar with Unix environments.

As it was your idea to go for such an approach, you have to lead this evaluation pro-active.

Best Regards, Thorsten.

[TK.]

Sidenote: here an example for a quite successful development platform for hobbyists. Arduino provides a single package which contains everything: IDE with editor and code upload tool, compiler, code library, a lot of templates: http://www.arduino.cc/en/Main/Software. Check also, how other users release new code modules.

Everything is open source, so nobody needs to re-invent the wheel, an adaption of the IDE to MIOS could be possible.

I must say that this is the direction I would prefer for MIOS: within an extremely short learning period you are able to do modifications in existing templates and upload the code.

Ah - even they are also using their own makefile generator, such an environment could hide the autoconf process very easily (the user wouldn't regognize it ;-))

Best Regards, Thorsten.

[audiocommander]

hi everyone,

Concerning the usability evaluation: I would propose a two-step approach: first ask two or three programmers of User Projects to migrate their source code, thereafter ask common users (in the MIDIbox news section) to rebuild the applications (it isn't required to own the appr. hardware)

Goal of the evaluation should be to find out, how much support people need to install and troubleshoot the toolchain, and how long it does take to get the setup ready for rebuilding an application.

If there's no time-pressure, I volunteer for setting up an installer for Mac OS X.

A binary installer for all the necessary tools and a nice Xcode template would be wonderful, wouldn't it?

The same could be done for eclipse on Windows and Linux

...of course only, once I managed to set up the environment by myself ;-)

Arduino provides a single package which contains everything: IDE with editor and code upload tool, compiler, code library

I'm not sure if Arduino's IDE is really the ideal to look up to: I mean, it's a nice environment, but the main reason for me not choosing Arduino (and instead going for MBHP/MIOS) was in fact its interpreted language and specialized IDE: I wanted to program in C with the IDE of my choice.

That's because I learnt programming with Lingo (Macromedia Director) and based on the experiences I made back then, I don't feel like I ever want to use a slow "specialized" lingo again.

Nevertheless, I agree, that installers are the way to go :-)

Best,

Michael

[ptitjes]

Concerning the usability evaluation: I would propose a two-step approach: first ask two or three programmers of User Projects to migrate their source code, thereafter ask common users (in the MIDIbox news section) to rebuild the applications (it isn't required to own the appr. hardware)

Agreed.

As it was your idea to go for such an approach, you have to lead this evaluation pro-active.

Of course. I'll make a call for that around the end of the month.

If there's no time-pressure, I volunteer for setting up an installer for Mac OS X.

A binary installer for all the necessary tools and a nice Xcode template would be wonderful, wouldn't it?

The same could be done for eclipse on Windows and Linux

...of course only, once I managed to set up the environment by myself ;-)

Cool! And no there is no time-pressure at all :)

I will work on my PCBs during January but will follow the forum everyday, available for any help. Also, I will complete the Wiki pages little by little...

I wanted to program in C with the IDE of my choice.

[...]

Nevertheless, I agree, that installers are the way to go :-)

I could do an ebuild for Gentoo Linux. Also I never looked at the RPM and deb package formats. But I guess I could do that quite easily too.

As for Windows, I don't know if there would not be a licensing issue if we want to provide Cygwin as part of a complete package... Anyone knows about this ?

Best regards, Didier.

[TK.]

I wanted to program in C with the IDE of my choice.

A prepared IDE is the perfect choice for doing the first steps. Especially for people who just want to customize a header file, rebuild and upload the .hex file.

To remind you: thats 99% of all MIDIbox users, and we should focus on them instead of a minority of programmers who will find their way anyhow.

It could be part of MIOS Studio, or (which would go into another direction) a completely prepared Eclipse setup, which seems to be popular these days.

Of course, files can always be edited outside the IDE, thats normaly the way I'm doing, even when I'm using IDEs for other microcontrollers. But whenever I explore a new micro family (and I did this often in the past), I prefer to setup a project with a simple frontend tool before going deeper into the topic - just because it's much faster and less error-prone (especially when you get direct access to help pages for each particular parameter and option).

As for Windows, I don't know if there would not be a licensing issue if we want to provide Cygwin as part of a complete package... Anyone knows about this ?

The HiTOP IDE (which I mentioned before) doesn't install a complete Cygwin environment, instead it uses the cygwin1.dll (POSIX emulation library, 1.2 MB). I'm not sure about the licensing here, but it seems to be a way to call unix commands within a unix-style file system.

Best Regards, Thorsten.

[stryd_one]

What?!  ;D

I'll volunteer to try the windows one out, if TK decides that it is the way of the future, and when there's a little documentation for those of us not *nix-savvy :)

[ptitjes]

Hi S1,

I'll volunteer to try the windows one out, if TK decides that it is the way of the future, and when there's a little documentation for those of us not *nix-savvy :)

I wanted you to set up that doc out! :)

I don't have any Windows station at hand... so you might like to give it a try by yourself :)

[stryd_one]

Well there's a problem... I would love to help out, but I've no idea what's going on. Tonight I have tried to make sense of this thread, and I quickly get lost in the mysterious unix commands and technobabble. You said earlier:

I don't think that this is so unix-oriented.

And I can tell you now that it definitely is. I'd have to learn more than I want to know about unix, just to understand this thread, let alone the actual implementation of the idea. You're way too relaxed with unix and not thinking from the perspective of someone who doesn't know how to use it at all - like me... actually, most users know less about windows than I do about unix, and I can't make sense of all this, so I hate to think how they'd go.

A good example was that you said:

You only have to do "./configure --help=short" to have a clear and documented view of what build features a particular package has.

Well, it's not as simple as you imply, because first, users have to learn that such a command even exists. This is what I mean by you being too relaxed. You're not thinking from the n00b perspective. n00bs don't even know that they can type 'configure', let alone that option, let alone know what it will do when they do type it.

For this approach to be really useful, someone needs to write a manual of the available commands and all of their options, and I don't mean the man pages ;) Even then, the user has to read this manual and refer to it, and it will probably be very confusing for most people and time consuming anyway.

I think, not taking into account the technical pros and cons, the existing way of, 'edit a well commented text file, save, and type make', is a good way to go, simply because it's easy for anyone to understand.