Building the firmware

[bryn.parrott]

Hi @victor @oscgonfer

I decided to try building the (standard, as cloned} firmware in preparation for building some of the new code I’ve written to support the anemometer/raingauge version of SCS. It is just to get the hang of it. and: That a make will download from Git all of the dependencies that will allow the code to be built, and remove the annoying error squiggles pointing to missing dependencies. This will then clear most of the 700 odd errors that I see in the coding window showing only actual coding errors.

I am running the process directly against the clone that I made of the FablabBCN repository of smartcitizen-kit-21 repo. It is unmodified version of the code.

And I have run into trouble at the first hurdle.
A basic make does not seem to work. I am obviously doing something wrong, or my environment is not set up correctly.

Looks to me like I am missing some files or folders. ‘requests’
(and its true, there is no folder named requests.)

Environment installed:

1. : Toshiba Laptop running Windows 10
2. Python 3.8 32 bit and Python 3.7 64 Bit
3. VS Code with Platform IO 2.3.3
4. Git 2.33.0.2
   Please can you assist.
   I have attached a screen shot of the feedback I got:
   
   

Many Thanks,
Bryn Parrott

[bryn.parrott]

OK, some more on this. I carried on further as seen in the next screen dump:
It seems that the file sck.py is having trouble with the following line of code:
self.paths[‘pioHome’] = [s.split()[1].strip(’,’).strip("’") for s in subprocess.check_output(
[‘pio’, ‘run’, ‘-t’, ‘envdump’]).decode(‘utf-8’).split(’\n’) if “‘PROJECT_PACKAGES_DIR’” in s][0]

wherein the file its trying to open is not found. I cannot tell what file it is; therefore unable to easily correct this problem.

Its looking to me like this build process has not been debugged in the Windows 10 environment. Unfortunately I do not yet have another environment to work on; because Apple have not yet released the macbook-pro 16 inch that I am waiting on.

How can we proceed ?

[bryn.parrott]

The way I have got round this problem (as a work-around) is to arduously clone and copy all of the repositories for dependencies into /.pio/libdeps

This has left me with just 3 dependency errors:
sam.h
variant.h
ArduinoWrapper.h

These files can be found within \sam.platformio however platformio does not find them; and I am kind of hoping they would be found automatically during a build; the reason for conducting the build process mentioned in 2 earlier posts.
Its possible the errors might be fixed by an edit of platformio.ini but everything looks ok in there; I am uncertain what should be changed.

[cyrille.mdc]

Hi @bryn.parrott ,

I’m working from linux so adapt as needed.

You’ve probably done this, but did you clone the repo’s recursively?
For my repo cloning I did this:

git clone --recursive https://github.com/fablabbcn/smartcitizen-kit-21

The other issue I ran into was setting up the path for platformio - does the pio command work?

You’re likely way beyond the basic roadblocks I encountered, but perhaps not.

Cheers,
C-

[bryn.parrott]

Thanks Cyrile for your response.

Yes; cloning of the repositories recursively is one thing I did as it’s mentioned in the documented instructions.

The pio command does not work for me; which is a problem because it’s needed to build the firmware for the pm board; something I need to do.
I have a second pm board for my project that has two uart ports used for external anemometer and rain gauge sensors. The code has been written but not tested and debugged because it has not yet been built. However there is some hope I can build it using platformio directly. A second approach might see it work using Arduino IDE. There are no 3rd party libraries only some .h and .cpp files so it might just work with Arduino and set to Zero board.

At this early stage I was just trying out the standard build on Unmodified code to check that it worked. The main data board has the python build scripts so I thought it should be an easy first step.

Cheers and thanks

[victor]

Hi,

You are right, we haven’t test building the firmware in Windows OS. At this point my experience in windows is almost nonexistent, and for us, it is too much extra work to maintain the full tool chain in every OS, sorry.

As far as I understand, your problem now comes from not having the pio command in your path. Have you tried following platformio documentation on having the shell commands available on Windows?

On the other side, building the firmware on Arduino IDE is not supported, and setting the board type to Arduino Zero will not work since we need our own SCK2 variant to define pin usage.

If you don’t mind using Linux, you could easily set up a development environment on any old computer.

Hope this helps!

Cheers
Victor

[bryn.parrott]

Hi victor

At the moment: pio commend via windows power shell does not work. I need to investigate why; but solution may be as simple as adding a string to the path command.
Lack of pio command currently prevents me from building the firmware for 2nd pm board which has no python script to help.

But that’s not the only issue. The python script for main data board also does not work; and the (current) cause seems to be 4 lines of code on the SCK.py script. Since the code fails at that point other similar issues may be lurking once that problem is solved.

I need to be able to compile the firmware for main data board to include integrations for: pm board #2;
Anemometer; Rain gauge; scd41 co2 gas sensor; 2nd Groove oled.
I am part way through coding for those integrations at the moment; there is a little way to go yet; when it’s done then it’s possible to test the whole system end to end.

I feel obliged to do my own coding for the above simply because it’s obvious you guys just do not have the resources to do it for me in any reasonable time and maybe it’s unreasonable to expect you to do it too.

Tell me it’s not true ! As it happens I am happy to carry on with it and polish up on my ancient c/c++ skills.
My experience as a IT contractor serves me well in analyzing other peoples code that is often undocumented in the fine details like we find here.

I currently have only a windows 10 laptop to work with. I am waiting Apple to release the new MacBook Pro 16 with m1x processor likely in October ; and maybe moving my dev environment over to it will allow everything to work properly.

Meanwhile I would appreciate a little help; as I find myself having to analyze almost every line of code and find there are some not so obvious things going on. For example what rules apply to adding new sensors.? There are two structures of interest.

1. The array in Sensors.h does not include I2C address.
2. An array in auxboards.h has a list all of the I2C addresses for supported devices but no apparent key into the sensors.h array and At the moment I have not found how they are tied together.
   I am not critical of the code; it seems to be well crafted but my current task is to use surgical skill to graft in an extra liver without the benefit of “greys anatomy”….

Cheers
Bryn

[bryn.parrott]

OK; Further to the above.

I investigated, and found that the CLI extensions to platforio GUI are present; and the path variable has been set.
What is not working is the ‘powershell’ command executed inside platformio does not inherit the windows system environment.
When powershell is executed in windows directly; it inherits the path correctly; and the pio command works as expected; I was able to build the firmware for PM Board #1. It compiles for the board ‘zeroUSB’

Victor advised that ‘sck2’ is the correct board to use; so I had changed the platformio.ini file for my PMB 2 firmware and found that it cannot identify the sck2 board. vis : Error: Unknown board ID ‘sck2’
So, I will change the platformio.ini file back to the way it was and hope that it works.

I might even go so far as to add in some debug to my code and connect up the board to my PC and see if the code appears to be working. In debug mode; without connection to the aux bus with an active data main board; the I2C part will not be functional; however it should at least run the basic ‘begin’ code for each of the two sensors.; and get some readings every 30 seconds or so from the anemometer and rain gauge. Not much wind or rain in my living room however.(plenty of hot air).

Cheers

[bryn.parrott]

Hi @vistor.

OK. At this point I have reached a point in coding where I think I have done all that is needed to integrate Anemometer and Rain Gauge on s econd PM Board; and SCD41 as a near-clone of the SCD30 CO2 sensor ; just connect to the SCD41x library. There are no error squiggles issued by platformio.

I have been able to build the firmware file for the second PM Board successfully; but have no (so far) loaded it onto the board.

I would very much like to compile the SAM code so as to complete the debugging; then move to load the code onto the boards. this requires the make.py script to work.

Whilst you say its too much work to maintain the tool chain in Windows :
a) the documentation indicates use of windows is a possibility; it gives instructions for windows environment.
b) as recently as April this year, Oscar has made changes to suit windows. I assume this means it was tested and found to be working at that time; otherwise the code would not have been committed to the repo.
c) Its unreasonable for you not to provide support for the tool chain under windows; as I proceeded with the project on the implicit understanding this part could be done; the documentation tells me so. Not to provide support leaves me high and dry; I sit here looking at my system; almost finished needing firmware that I have coded but not able to compile.
Perhaps you can see things from my perspective; I have certainly made allowances for yours. I am aware of your lack of resources and thus slowness to respond.
d) Even if you have no dedicated windows PC’s available; you are in fact able to provide a windows environment via a virtual machine such as VMWARE operating on a Mac or Linux environment. How was it done in the past ?

I have no knowledge or prior experience of Python; indeed I do not know what platformio is doing under the hood either. To me its just another text editor. I do not know what python is contributing to the process of compilation that PIO itself cannot do; or indeed, why arduino ide cannot/should not be used. The learning curve is rather steep in my case.

[bryn.parrott]

@victor @oscgonfer : I can not force you to support Windows, except to vent my disapproval of your unheralded decision to stop supporting it in mid-project for me.
So, in order to have some forward progress; I decided to try an unthinkable option; to ask my partner to loan me his MacBook Pro 13 inch for indeterminate period (until I can order my own).
So, I did that; set up myself as an administrator; then proceeded to install the tool chain; Python 3; Git, GitKraken, Visual Studio Code with Platform IO; pip (upgrade); requests; serial. I also installed an update for the Mac; it is running BigSur 11.6.

And then I tried to execute the make file. It failed. See attached screen shot.
At this point again; I am flying blind; I am not used to the mac environment.
The laptop is intuitive, but ‘different’ than windows. I know nothing of Python; so I am unable to problem-solve fluently. The error reports are numerous, I do not know where to begin.
Clearly it is having trouble finding paths.
Prima Facie the code is broken in this environment; needs some adjustment.
Here is the screen shot.

[bryn.parrott]

OK, again to try to make some progress I looked at a couple of things.
I determined that the ‘pio run’ command works by building the firmware for smartcitizen_kit_pm_board. This involved (similar to windows) adding a path statement to the system /etc/paths file.

So, I tried using the ‘pio run’ command against the ‘smartcitizen-kit-21’ project.
Initially, the program satisfied a heap of dependencies by downloading from various repositories and loading them into the .platformio folder. It had trouble on a couple of them; so I just bumped up the version to force it to work.
However ultimately, the pio run command did not work; it got hung up finding the file ‘variant.h’ required by several of the core library files. I do not know how to solve that issue. ( i looked at various ways).’

So, then I again tried to build the project using the python make.py file.
This time it got different errors than before; the screen shot below shows them. There are many similar errors.
basically there is a dependency on a library called ‘RadioHead’ and within that library there is a file ‘hardwareserial.h’ of which there are two versions; so the compiler says ‘too hard’ and stops.
You must have come across this; perhaps there is an easy fix you can suggest. My approach is to determine the correct version of hardwareserial.h and delete the other from the library. Is there a better approach ?

[oscgonfer]

Hi,

This is good progress! Considering where we were some months ago thisSome comments here:

• We do support Windows as long as the tools that we use for our projects also support it. This means that by using platformio for the compilation and flashing of the SCK, we intrinsically support Windows by default, since platformio is written on python and python works on everything. What we should improve on is specific instructions for people that are not so used to these things

• If there are not so many instructions on how to build the firmware in Windows, is because we do not have Windows machines, but I have personally put a lot of effort on a guide for making the compilation process available on Windows too, by using the tools that @victor initially created in make.py and the sck class.

• The problems I see you are facing are related to generic python installation. In this last screenshot, there is a problem with pyserial, which apparently is not installed. You would need to do pip install pyserial==3.5 - You can see it in ModuleNotFoundError: No module named 'serial.tools'.

The rest seems fine to me. I have added a requirements file to the tools repository so that you only need to do: pip install -r requirements.txt. You can simply pull from the tools submodule or the firmware one (by doing git pull --recurse-submodules)

[oscgonfer]

Adding more to this:

• requirements file added and tested: https://github.com/fablabbcn/smartcitizen-tools/blob/905d3709103b07834c36c41957e9fa673220ee41/requirements.txt
• docs updated: Edit the firmware - Smart Citizen Docs

[bryn.parrott]

Thanks @oscgonfer for your response.

Concerning windows support I am happy to test it in order to determine issues and efficacy of any fixes; but for my own purpose on the SCS wind&rain project having spent the time migrating to Mac Book pro I intend to stick with it; I will have to do another migration when Apple come to the party and release the one I want to get. My partners laptop is ok and blindingly fast but the tiny 13inch screen is too limiting for me.

I have yet to move over my code base from windows to Mac; but there’s an app for that so it should be ok.
I will only do that when the build process works on original codebase.

We can talk later about whether my changes should be on a branch (of 0.9.8) or keep it separate “island” development.

I am presently away from home looking at moving to Taichung in central Taiwan; will return tomorrow and can test the fix you suggested.

That’s good work you have done on the docs. I have in mind to take the learnings from this exercise and turn it into something others might use to save them the same pain as I have gone through the last couple of days. So that will be something like.

1. Setting up the environment on Mac and windows. Extending the doc you already wrote.
   1a discussion of how platformio works and how to make best use of it.
   1b Cloning of repository; copying of local clone to make a local workshop copy. Making of a branch; making a written submission of proposed changes back to Fablabbcn team.
   1c discussion of the functioning roles of various (pairs of ) files making up the SCK data board system. Like a map.

2. The work I have done on integrating sensors in two situations
   (a) a device on the AUD bus
   (b) a serial uart device connected to a pm board
   Discussion on the changes needed on

• sensors.h
• SCKAux.cpp and sckaux.h

Discussion on other kinds of changes people might want to make;

Discussion on what background knowledge and general method of approach is needed to competently make changes to the SCK firmware.

[bryn.parrott]

@oscgonfer
There do appear to be differences in default python modules included with Mac version compared to windows version.
Thanks for the information about pyserial; as mentioned I will fix that tomorrow and see what the next issue is after that. And repeat…… until it works.

[bryn.parrott]

Hi @oscgonfer

OK, I just got home, and made the installation of pyserial that you suggested. Then I executed > python3 build sam -v as before and attached 2 screen shots show the results.
It gets hung up on ambiguity between two different versions of hardwareserial.h contained within the RadioHead library.
This might not be a fatal issue; (because later it seems to carry on and find other issues) but certainly it causes numerous errors in the output and probably ought to be fixed somehow. can you suggest a way ? (or is my proposed fix; to cull one of the two duplicates) reasonable ?

[bryn.parrott]

OK, I have done some digging and found the following.

The error occurs when ‘uart.h’ *(part of the arduino distribution) calls for ::
#include “api/HardwareSerial.h”
But the compiler has other ideas. It decides that the ‘hardwareserial.h’ file it saw earlier included in RadioHead might be just as good. and gets confused as to which one is required. Now, uart.h is pretty specific as the the file it wants.
so… I will temporarily remove RadioHead from platform.ini and see what happens.

[bryn.parrott]

OK, That theory turned out to be wrong.
Removal of the radiohead library just caused other issues.
commands.h includes sckbase.h which wants rhreliabledatagram.h and rh_serial.h both of which are part of the radiohead library.

the hardwareserial.h included with the arduino distribution seems to be the issue.

in fact both of the hardwareserial definitions might be required …

[bryn.parrott]

OK, I decided to roll things back again. This is a sanity check. Its incomprehensible to me that the FablabBCN @victor would have been able to roll out 0.9.8 release having errors like the ones I am seeing. I must have done something wrong.
So, I rolled back platform.ini to the version included in the clone; and removed everything from .pio folder. Then execute the make.py script once again.
this (below) is the output I get. Its barfing on SparkFun_u-blox_GNSS_Arduino_Library because the version 2.0.7 requested is not the latest version (2.0.15). I previously fixed this issue by just allowing it to download the latest version. I checked the repository for this version; and its listed in the releases. I tried to convince GitKraken to download the correct version but it was not having any. So; I changed platform.ini to permit 2.0.15 to be used.

The same thing happened with https://github.com/adafruit/Adafruit_MPL3115A2_Library
which wanted version 1.2.4 but the downloader only wants to download the latest; which is 2.0.0. Again; I changed platform.ini to allow that to happen.

Those two changes allowed the downloader to proceed to get the arduino package libraries. Unfortunately; I see again that the downloader gets hung up looking for ‘variant.h’.

I know that this can be fixed by again changing platform.ini to remove version dependency from the entry (platform = ) ::
[env:sck2]

build_flags = !sh …/tools/git-rev.sh

platform = atmelsam@5.0.1

platform_packages = framework-arduino-samd@1.8.9

board = sck2

framework = arduino

lib_extra_dirs = …/lib

extra_scripts = uploadSAM.py

I also know that if I fix this issue by removal of version (5.0.1) dependency this will expose the problem seen earlier; an issue with hardwareserial.h

Therefore I need you (and perhaps @victor too) to please take a look at this and tell me how to
(a) get the correct (ie tested) versions of the 3rd party sensor libraries and
(b) get a version of arduino platform related libraries…
that will allow the build process to complete successfully.

I am using your tools on your preferred platform (mac) so as far as I am concerned it ought to work.
I am fresh out of ideas. I am hoping it’s an easy and quick fix for you.

[bryn.parrott]

Ok,
Concerning the issues downloading earlier versions of sparkfun gnash library and adafruit 3115a2 library. The version of python I am using on the MacBook platform seems unable to download this earlier versions. I am thinking it’s likely python is somehow set to always download the latest master if maybe it does not like the fact those versions of those libraries are not present in unbundled form.

Wheat ever is the cause; I suggest the solution might be to download and unbundled the correct version onto the Fablabbcn GitHub site (as has been done with other libraries) and point platformio to that location instead of the original authors repo.
Alternatively you might also just test the firmware using the latest version of libraries; but maybe you do not have the time to conduct the necessary regression testing.

Whatever solution seems appropriate to you…

*My solution should be to *
(a) run the make script with modified platformio ; this downloads libraries into the libdeps folder using current release versions of the two libraries.
(b) download and extract correct version of the two errant libraries over the top of the newer incorrect versions located in ‘libdeps’
( c ) revert the platformio.ini file to demand the correct version; which is now located in libdeps.(so it’s happy)
Then run the make file again; and it should now be happy and step over the version issues.
I tried that and it did not work. Python make.pl still tried to download the latest version.

It’s not satisfactory; but it’s a work-around; until you decide the best solution. After all this version of MacBook is using the latest iOS so it’s here to stay.

Meanwhile we have the 3rd issue concerning missing ‘variant.h’ which is something I am unable to fix directly, except to say it’s related to the version of platform ; and again this seems to be an issue with the build tool chain on the MacBook pro platform.
my previous fix(to remove the specified version ) only leads to later problems. Therefore as aforesaid I need you guys to look into it and suggest a solution.

Over to you…