Building the firmware

[bryn.parrott]

Hi @oscgonfer

OK, I just got home, and made the installation of pyserial that you suggested. Then I executed > python3 build sam -v as before and attached 2 screen shots show the results.
It gets hung up on ambiguity between two different versions of hardwareserial.h contained within the RadioHead library.
This might not be a fatal issue; (because later it seems to carry on and find other issues) but certainly it causes numerous errors in the output and probably ought to be fixed somehow. can you suggest a way ? (or is my proposed fix; to cull one of the two duplicates) reasonable ?

[bryn.parrott]

OK, I have done some digging and found the following.

The error occurs when ‘uart.h’ *(part of the arduino distribution) calls for ::
#include “api/HardwareSerial.h”
But the compiler has other ideas. It decides that the ‘hardwareserial.h’ file it saw earlier included in RadioHead might be just as good. and gets confused as to which one is required. Now, uart.h is pretty specific as the the file it wants.
so… I will temporarily remove RadioHead from platform.ini and see what happens.

[bryn.parrott]

OK, That theory turned out to be wrong.
Removal of the radiohead library just caused other issues.
commands.h includes sckbase.h which wants rhreliabledatagram.h and rh_serial.h both of which are part of the radiohead library.

the hardwareserial.h included with the arduino distribution seems to be the issue.

in fact both of the hardwareserial definitions might be required …

[bryn.parrott]

OK, I decided to roll things back again. This is a sanity check. Its incomprehensible to me that the FablabBCN @victor would have been able to roll out 0.9.8 release having errors like the ones I am seeing. I must have done something wrong.
So, I rolled back platform.ini to the version included in the clone; and removed everything from .pio folder. Then execute the make.py script once again.
this (below) is the output I get. Its barfing on SparkFun_u-blox_GNSS_Arduino_Library because the version 2.0.7 requested is not the latest version (2.0.15). I previously fixed this issue by just allowing it to download the latest version. I checked the repository for this version; and its listed in the releases. I tried to convince GitKraken to download the correct version but it was not having any. So; I changed platform.ini to permit 2.0.15 to be used.

The same thing happened with https://github.com/adafruit/Adafruit_MPL3115A2_Library
which wanted version 1.2.4 but the downloader only wants to download the latest; which is 2.0.0. Again; I changed platform.ini to allow that to happen.

Those two changes allowed the downloader to proceed to get the arduino package libraries. Unfortunately; I see again that the downloader gets hung up looking for ‘variant.h’.

I know that this can be fixed by again changing platform.ini to remove version dependency from the entry (platform = ) ::
[env:sck2]

build_flags = !sh …/tools/git-rev.sh

platform = atmelsam@5.0.1

platform_packages = framework-arduino-samd@1.8.9

board = sck2

framework = arduino

lib_extra_dirs = …/lib

extra_scripts = uploadSAM.py

I also know that if I fix this issue by removal of version (5.0.1) dependency this will expose the problem seen earlier; an issue with hardwareserial.h

Therefore I need you (and perhaps @victor too) to please take a look at this and tell me how to
(a) get the correct (ie tested) versions of the 3rd party sensor libraries and
(b) get a version of arduino platform related libraries…
that will allow the build process to complete successfully.

I am using your tools on your preferred platform (mac) so as far as I am concerned it ought to work.
I am fresh out of ideas. I am hoping it’s an easy and quick fix for you.

[bryn.parrott]

Ok,
Concerning the issues downloading earlier versions of sparkfun gnash library and adafruit 3115a2 library. The version of python I am using on the MacBook platform seems unable to download this earlier versions. I am thinking it’s likely python is somehow set to always download the latest master if maybe it does not like the fact those versions of those libraries are not present in unbundled form.

Wheat ever is the cause; I suggest the solution might be to download and unbundled the correct version onto the Fablabbcn GitHub site (as has been done with other libraries) and point platformio to that location instead of the original authors repo.
Alternatively you might also just test the firmware using the latest version of libraries; but maybe you do not have the time to conduct the necessary regression testing.

Whatever solution seems appropriate to you…

*My solution should be to *
(a) run the make script with modified platformio ; this downloads libraries into the libdeps folder using current release versions of the two libraries.
(b) download and extract correct version of the two errant libraries over the top of the newer incorrect versions located in ‘libdeps’
( c ) revert the platformio.ini file to demand the correct version; which is now located in libdeps.(so it’s happy)
Then run the make file again; and it should now be happy and step over the version issues.
I tried that and it did not work. Python make.pl still tried to download the latest version.

It’s not satisfactory; but it’s a work-around; until you decide the best solution. After all this version of MacBook is using the latest iOS so it’s here to stay.

Meanwhile we have the 3rd issue concerning missing ‘variant.h’ which is something I am unable to fix directly, except to say it’s related to the version of platform ; and again this seems to be an issue with the build tool chain on the MacBook pro platform.
my previous fix(to remove the specified version ) only leads to later problems. Therefore as aforesaid I need you guys to look into it and suggest a solution.

Over to you…

[cyrille.mdc]

Hi @bryn.parrott ,

You PMed me but I’ll respond here so that this can help others.
You mentioned that, as you state above, you’re having problems getting it to build SAM on your mac.

I’ll provide my notes but I have to say getting SAM building wasn’t too hard so this may not be much help:

I needed to set my path for pio correctly so that it uses the correct installation of serial.tools:

Note the order below - this caused me problems (for ESP but not SAM if I recall correctly but I may be wrong) (see here for more info):

export PATH="$PATH:/home/cyrille/.platformio/penv/bin"

Then I just built:

python make.py build sam -v

You should get/want a [SUCCESS] message.
There’s a bug where (noted somewhere) you may need to run the above twice to get the SUCCESS message.

I had to also update sam/platform.io to ‘point’ to the latest versions.

Unfortunately that’s it. Getting SAM to build was much easier than getting ESP to run because of my little path problem and serial.

[bryn.parrott]

Thanks Cyrille for your response;

The fix you suggested is something I did right at the start.

I implemented the path addition in /etc/paths and it needed to be more explicit.
Before I made the change; “pio run” command would not work; after I made the change it worked; thus I consider it effective.

The issues I am finding have occurred during execution of the make file;
Two type of error:

1. The make process appears to download only the latest version of libraries instead of the @ version specified including platformio.ini then it discovers it does not meet the @ specification and barfs. It does this on only two libraries; the MPL3115a library and the Sparkfun ublox gnss library.
2. After that it finds problems finding parts of the Arduino platform library tools such as variant.h; there is also an issue with ‘hardwareserial.h’ found in both the Arduino distribution and the Radiohead library; causing a conflict.

I had similar issues when trying all of this on the windows10 platform; the reason I moved over to Mac platforms.

I am starting to think it ‘has it in fit me’ (lol). Or maybe I’m just stupid.

At this moment I am thinking possibly the underlying issue is the versions of tool chain files (Python ; Git; platformio) but that’s clutching at straws;
Potentially the fact that Apple include a version of python on the platform might be an issue; the latest python I installed must specifically be called using ‘python3’ on the command line.

[victor]

@bryn.parrott

The make process appears to download only the latest version of libraries instead of the @ version specified including platformio.ini then it discovers it does not meet the @ specification and barfs. It does this on only two libraries; the MPL1113a library and the Sparkfun ublox gnss library.

Hi!
Yesterday we realize this was happening, we fix it on this commit please pull the new code and try again, hopefully this will help you!

[bryn.parrott]

Wow !

thanks for your help. I will ry it right away. Lets see if GitKraken is worth the money I paid for it.

also I found that I have two versions of Python installed:

When I execute the make script; I type: python3 ./make.py build sam -v
thus hoping to execute that latest Python3 version.

Please can you verify that:
a) Python 3.9.7 is optimal to execute the make file.
b) If a python command is executed within the makefile (or sub-scripts) that it will find the correct version (because /usr/local/bin is first on the $path)

These are my assumptions anyway; am I correct ?

[bryn.parrott]

Hi @vicor @oscgonfer

That change to platformio.ini did indeed fix the problem with getting correct versions of 3rd party libraries.

This leaves me with the next problem, see screenshot.
the make file is unable to locate ‘variant.h’ which is usually part of the distribution of arduino board packages etc. This file is referenced many times during the build, of which only one is depicted. This same problem was seen when I attempted the build process on the windows platform earlier.

[victor]

Hi,
We have a PlatformIO custom board called sck2 that has its own variant files to define the different ports we use. Every time you build the SAM firmware a script is called that makes sure that the variant exists and hasn’t been overwritten, it uses the PlatformIO core_dir defined in the platformio.ini file.

I’m not sure, but it is possible that something related with what I have described doesn’t work properly in a Windows environment. Please check that you have this directory (.platformio) and that the board SCK exists in there (it should be in the same path mentioned before).

Hope this helps!
Victor

[bryn.parrott]

Thanks @victor for the pointer; I was aware that the folder existed; but not how it hangs together.

I checked; and found the sck2.json file within the ‘boards’ folder.
within the file it points to ‘frameworks’ keyword; with the value ‘arduino’ vis-a-vis:

The variants file is located under folder ‘framework-arduino-samd/variants’
Its probably a pretty dumb question and forgive me please for asking it;
Also I note that this file was last edited 13 months ago; and obviously its working on other platforms OK; but I wonder if maybe :
“frameworks”: [arduino]’ is really an alias for ‘frameworks-arduino-samd’
but otherwise I do not see how it could be that the variants.h file is to be found by the build system. Maybe its looking in the wrong place.
All I do know is that ‘delay.h’ wants it; and unable to find it:
.platformio/packages/framework-arduino-samd@1.8.9/cores/arduino/delay.h:23:10: fatal error: variant.h: No such file or directory. (7 times)

I remember from (many) years ago (vaguely) there was a diagnostic utility program available for developers under windows; its function was to monitor filesystem activity by a process; and log the files that it attempted to open or ‘touch’.
Do you happen to know if there is such a utility available for the MAC iOs environment that I coukd use to see where it is looking for ‘variant.h’ ?
That would be something like ‘fsmon’; {https://github.com/nowsecure/fsmon}
is it useful in this context ?

[bryn.parrott]

@victor @oscgonfer [ please read until the end of this message]
I tried installation of a tool called ‘fsmonitor’; It displayed a lot of information; but none of it useful to this purpose of discovering whether ‘variant.h’ is touched during compilation.

OK; So… I tried another pathway; (
I cleaned out the .pio and .platformio folders (except for framework-arduino-samd)

Then executed the make script. (and it got the same result as before - cannot find ‘variant.h’) (I am not surprised by this.)
then looked at the folders that were installed into .platformio

I notice that there are two folders for framework-arduino-samd.
one called framework-arduino-samd
the other called framework-arduino-samd@1.8.9

The difference between these two folders:(?)
- framework-arduino-samd@1.8.9 has ‘extras’ folder
- it has variants folder that does not contain a sck2 fubfolder and hence no variant.h for the sck2

-framework-arduino-samd has:
- NO extras folder
- variants folder contains subfolder sck2

It seems possible that maybe the make process looks for ‘variant.h’ within the framework-arduino-samd@1.8.9 folder; and does not find the sck2 folder so it gives up ‘file not found’; But why are there two folders ?

So… I tried one more thing. I copied the folder ‘sck2’ from one folder to the other.

Then I ran the make file again. Here are the last two lines of output:

We found 197.04MB of unnecessary PlatformIO system data (temporary files, unnecessary packages, etc.).

Use pio system prune --dry-run to list them or pio system prune to save disk space.

Wrote 446976 bytes to /Users/brynparrott/Documents/Platformio/projects/smartcitizen-kit-21/bin/SAM_firmware.uf2.

.

Lo and behold; the make process has executed from beginning to end successfully.
It gave up a few warning messages; but completed and delivered a firmware file. I have no idea if it works or not. I will look into a few of the warning messages to see if they look important; before loading it into my scs data board.

For you to consider is the cause of this issue and whether and how to correct it.
Maybe it just needs a documentation fix (simplest) or …

[bryn.parrott]

@victor @oscgonfer

JFYI

I took the same learning over to my windows 10 PC.

I applied the fix you made to platformio.ini; however I found that when make.py is run; that I now get another different error; the script is unable to run the ‘sh’ command that would allow it to execute the git-rev.sh command script;
Now; It was previously able to do this (I do not know how); so I do not know what has changed to stop it from working. Since the ‘sh’ command never has existed on windows10 the make file must have a work-around; and that is no longer working .for some ungodly reason.

I checked the ‘pio run’ command and found that it also has the same issue.

At this stage; like you; I am inclined to just drop it; say that the build process does not always work on windows 10 and leave it. I have another option…

Bye the bye; I checked previous partial builds on windows; and found that the problem seen on the mac; namely that the folder ‘framework-arduino-samd’ is duplicated; not completely merged with framework-arduino-samd-1.x does not appear to occur when the windows build process can execute.

[victor]

Congratulations!

In a clean clone of the repository in Linux, after building I only have the framework-arduino-samd not the framework-arduino-samd@1.8.9 folder. I don’t know why is different in Windows, when I have some spare time I will look into it and try to find a permanent solution.

To be sure everything works as expected you should also copy the second file (SAMD21_I2SDevice.h) indicated in this script. It fixes some problems with the I2S bus.

Cheers
Victor

[bryn.parrott]

Thanks @victor for the heads up on that file.

Next time I do a build I will check and make the change suggested.

@oscgonfer
Right now I have moved over to the physical task of wiring and final assembly, to be ready to “power up the beast”

This will entail online registration of the device; and I plan to connect via the 4G gateway wifi as well.

Also to this same end I have sent Oscar a link to a Google docs file that contains technical details of the Spec Sensors gas sensors. This includes:

• sensitivity code,
• serial numbers
• I2C mapping for each of 5 gas sensors.
• Temperature compensation data for calibrations

Of course I have only the 4 port ADC right now so one sensor (O3) will not function.
It’s looking like the 4 port ADC you included will be OK in fact I don’t see why you said I would find it unsatisfactory; unless of course it does not work. We will soon see I guess.

If it’s true then I am only short the one 2 port ADC and the box for the 4G gateway.
XxAnd a power cable for the 2nd PM Board…xx
Update: I purchased a micro-usb cable and cut it in half; made myself the power cable.

Meanwhile I will soon be able to power up and then load the firmware and start testing.

Cheers

[bryn.parrott]

Ok @victor @oscgonfer

The latest issue in this saga is that I am having trouble on MacBook Pro platform reliably recognizing the (2nd) PM Board plugged in for firmware loading and monitoring of debug.

The cable used to connect the PMB is a standard micro-usb meaning it has uUSB at one end & USB A at the other. There are no such ports on the MacBook; it’s connected using a dongle that plugs into both USB-c ports and provides 2x USBc and 2USBA ports.
I am thinking that this method of connection might be flawed perhaps the RTS/CTS or DTR signals that are needed to perform automatic reset of the PMB following firmware upload might not be getting through; thus causing the PMB not to behave correctly.
Secondly there is the issue I am unable to monitor SerialUSB debug stream to confirm correct firmware operation.
Thirdly it looks like firmware upload is occurring at default speed of 1200 baud rather than the set speed of 115200 baud.

The second problem is by far the biggest concern.

I went to local electronics shop and purchased several micro usb to usb-a cables and one usb-a to usb-c adapter; to try and eliminate the dongle from the equation. Unfortunately the one I bought usb-c plug is incorrect and does not securely plug into the MacBook.

So now; I am attempting to purchase a direct micro usb to usb-c cable hopefully with all the 8 wires connected through thus allowing rts/cts/dtr lines to pass through and perform reset when required.
This is not available in Taiwan in regular electronics shop; so it will need to purchase from US; some more delay. And I am in process of moving house which complicates deliveries.

Finally I looked at what is happening with the 10 pin connector: when I touch one pin with an open ended wire it seems to trigger a reset. I connected oscilloscope.
The thing I noticed there is quite a lot of random electrical noise in the environment even when GND wires are connected. I think that the open ended wire is acting like an antenna; and the signal is strong enough to trigger the reset.
Further to consider: the 10 pin connector is designed for use to connect the HWD device Atmel ICE or similar; on this same port on Data board; a ribbon cable is connected that goes to a reset button that is accessible from outside the enclosure.

I have in mind to borrow the reset button from the data board and connect it to PMB temporarily to use as reset. So the SCS will again need to be disassembled…(now that I finally got it all back together again….irony)

Also; I checked the firmware code and found some errors; re-implemented “pins.h” so that coloured leds can work and pin-naming for the two serial devices can be consistent.
But those bugs would not likely cause SerialUSB not to work for debug output.

So; I am thinking perhaps I should put together a simple Arduino sketch to establish that SerialUSB and reset function can work. Then work up to PMB firmware piece by piece. Back to basics….

You can see where I am going

If you have any suggestions on any of these items I would love to hear them; might save me some time.

Cheers

[bryn.parrott]

OK,

I went back to basics; I used Arduino IDE to build for the blink sketch; then elaborated to exercise the RGB LED. Then I added code to output debug on the ‘Serial’ port which did not work; changed it to ‘SerialUSB’ and the debug happened.

Arduino IDE was able to upload firmware and monitor the USB serial port in terminal window without any problem whatsoever.

So, then I moved the test code into the project folder and renamed the file “pm2_driver.ino” to pm2_driver.xxxino"; I also removed the manually configured port configuration from platformio.ini; and restarted VS Code.

I found that the file was built without any errors; and uploaded OK after a reset. I was even able to get it to monitor the serial output although it did not switch from upload mode to monitor mode quickly enough to catch the first few lines of output.
I tried the ‘reset’ button and found it does not work reliably; I am wondering if I have it connected in correct orientation. Lacking a plug location key; it can be connected in two ways;

So; I can go on now I think and try to debug the actual code;

[bryn.parrott]

@oscgonfer @victor

So now I have ordered an Atmel ICE debugging kit. Maybe its not essential; however I am sure that if I had ordered one in the past this issue could have been solved quicker.
I also ordered a I2C Driver from CrowdSupply which should be useful; in particular to help with testing correct functioning of the AUX I2C bus. (because) I have a concern there may be too much current being drawn from the I2C master (the Data Board).
I am not sure of what remedies are available if thats the case. I might need help from Oscar is it turns out to be the case. The main issue being that (I think) one or more Sensor boards have too low a value for pull-up resistors; but the value is not designed to be easily changed because they have SMD resistors and no jumpers.

Although its tempting to blame the tools e.g. PlatformIO, I feel its most likely that the main problem lies in my code; and my present target is the SerialPort definitions. I might be able to use Atmel Studio to investigate this more closely. I note PlatformIO can also work with Atmel ICE ; I am unsure what level of functionality it provides.

Getting back to the present issue; I am thinking (speculation) that my code has clobbered one of the lines associated with USB Port and the code on GPIO port has my laser sight guide on it. I do not have the time right now to look into this because I am shifting house and everything including the project kit must be packed up.

[bryn.parrott]

@oscgonfer @victor
OK; I have finished all the relevant unpacking, set up my new workspace and today is the first time I have been able to work on the SCS project in a few weeks, so I have a clear head.

I determined first to recheck my work on assignment of the sercom ports for the Anemometer and Rain Gauge.
I started with the Microchip Documentation for SAM D21/DA1 Family; looking specifically on the big table of PIN Multiplex on Pages 30-31.
I made a reduced list of all the pins and which sercoms & pads could be assigned to them.
I then moved to the circuit diagrams and determined which PINS are wired out to which physical ports.
Finally I looked at ‘PINS.h’ file in the repository code; plus ‘variants.h’ in order to determine which sercoms are actually used by all 6 serial ports.
I have the following (check me):
Sercom 5: (Pins 31,32) I2C
Sercom 4: (Unused)
Sercom 3: (Pins 33, 34) USB Port
Sercom 2: (unused on 2nd PM Board) (PMS socket A)
Sercom 1: (Pins 45,46) SWD Port AND (Pins 27,28) Uart Port.
Sercom 0: (unused on 2nd PM Board) PMS Socket B)

Along the way; I discovered one possible reason why I have made errors in the port assignments:
In the PINS.h file we see entries like:

// RGB Led pins
const uint8_t pinRED 	= 26;		// PA27
const uint8_t pinGREEN 	= 25;		// PB3
const uint8_t pinBLUE 	= 13;		// PA17

Now, I had assumed that the {label = number} correspond to physical pin numbers on the SAM; but my assumption appears incorrect.
The coloured LED wires are (according to the circuit diagram) are connected on PINS 48, 39 & 26 corresponding to Ports PB3, PB23 & PA17 ; So it looks to me like either the circuit diagram or the PINS.H file are incorrect.
And yet the code works (for the first PM Board) !!
Please can you enlighten me on what is going on here !!

My current proposed solution is to use the following assignment:
“ADC” port ADC0 & ADC 1 Pins 9,10; Ports PA4, PA5 assigned to Sercom 0 (ALT) Pads 0 & 1
and
“GPIO” port GPIO0 and GPIO1 Pins 21,22 ports PA12, PA13 assigned to Sercom 4 (alt) Pads 0 & 1.

This I hope will ensure that SWD port will remain available and no existing port will be ‘clobbered’ by my code.
It seems to me ironic that I am unable to actually use the designated UART port because sercom1 is used by SWD and sercom3 is used by USB and those are the only two sercoms that could work on PINS 27/28 PA18/PA19
I grant that some of the issue might be mine; that I do not know how to implement sharing of the sercom pots.

Anyway; I am proceeding. I will make the required code changes for this; also look at the issue advised by @victor a few weeks ago concerning the variants file. This first part of this will be to create a ‘clean’ working code base.

Cheers