Back to Smartcitizen.me

Building the firmware

Okaaay,

so I decided to jump in and do what seems plausible.

I have downloaded one of the JSON files sourced from:Here

working in the [Sensors] section only; I have removed entries for sensors that are not present in this particular instance of SCS.

I have added entries for new devices such as SCD41; Wind, Rain, (etc).

While I was at it; I decided to go ahead and include the EBS160/EBS210 devices also. I can do that because I have curtailed the 2nd OLED because it just does not appear to work; and this leaves me a spare Mux Channel.
So, I have now also fully integrated these two devices; giving another 6 reading types.

The ENS160 device is a VOC sensor; that has improved specification over the CCS811 device. It also gives usable data after a delay of only 3 minutes. I was gifted an evaluation kit for this device in mid-2021 and the eval kit comes with both ENS160 and ENS210. The ENS210 is a temp + RH sensor; highly accurate. The ENS210 device is read during a reading of ENS160 and the Temp + RH values are fed into the ENS160 used for calibration; making it more accurate.

After I made the integration for the two devices I was happy to find that the code compiled correctly first attempt and that readings became available immediately. Such was the experience gained during earlier work.

I will place the modified file into its own repository within my GitHub, and share it with you.

I will look and see if its appropriate for me to directly link it to my SCS V3 on the platform, but otherwise I now await for you guys @oscgonfer @victor to do your thing and stitch this together.

I actually need it to be done; to eliminate as possible cause why a lockup occurs when wifi config is first loaded. And indeed, I would like to see the various readings appear on the platform.
It matters not (at this stage) whether the GAS concentration readings are based on Alpha-sense calculations or SpecSensors (if you have not yet finished that task). If I can see Gas concentrations (even if incorrect) it will demonstrate that the data flows through from end to end; and it’s another milestone. I can move on to other things whilst you finish the Spec Sensors piece at your end.

I will share the repo with you in a little while…

Thanks and cheers,

Hi @Bryn,

Sorry, but we are really saturated these days.
Steps for this last aspect to work:

  • First we need to include the additional sensors in your unit in the platform, so that they can be ingested. We need to decide how this is done, becase they are a lot of sensors and we need to decide which ones are to be sent and not, so anything from these lines will be ignored in the platform unless the sensors are there. I suggest going with Wind Speed, direction, one or two of rain and that’s it. The reason for this is the json package size. If we see there is no issue, we can maybe later on include the other metrics, but I think it’s too specific to your case (both current and ENS160/210) values
  • Secondly, the IDs that the platform give use would need to go in Sensors.h, which I see you have properly added in your version of the firmware but they might not be the same we get. We’ll see
  • Third, the sensors part in the blueprint is only for data download purposes, all the calculations are done according to the metrics part. We need to now identify: (a) which things do we need to calculate in the platform, (b) the channels needed for those calculations, (c) if those require calibrations, and (d) if there are IDs for those calibrations we can use. (a) and (b) will go into scdata repo. (c) I assume is only for SPEC sensors, (d) we have to put in the calibrations.json file. Maybe you can use the example for alphasense sensors in here and add the file to your repo, with the SPEC sensor IDs and calibration values you have.
  • Finally, we need to add a hardware descriptor file in the hardware folder that defines your unit. The name, for instance, SCAS2100BP.json, will be what we put in the postprocessing field in the kit edit view (in this case, SCAS2100BP will be what we put there)

So, if we agree that these metrics are fine for now, we can go ahead and add them to the platform. Then we will send you the IDs and we can move on.

Regards

Thanks @oscgonfer for your response.

I’ll go through them one by one.

  1. Limitations in the number of readings that can be sent. Can I assume this limit is based on MQTT packet size; something less than the TCP packet size of 2500 bytes.(?). Is it possible to make a calculation of (roughly) how many readings will fit inside a packet? The calculation can provide a target.
    My mind immediately went in the direction of possibly interleaving readings but sending them through more often; say every 30 sec instead of 60 sec.
    Yes there are some sensor readings that I feel may be limited use or duplicated. For example the SCD41 has its own temp and RH readings; but documentation says they are thrown off by temperature effects of the CO2 sensor. Thus pretty much useless.
    Also I am looking at the readings from twin PM sensors and wondering if they can be aggregated on the device instead of sending both sets to the platform. Is it simple averaging that’s being applied ?
    Really only one set of dust figures is needed.

I agree that the INA readings do not need to be sent to the platform; since they are just a convenient way to keep track of power drain on the battery. It would be far better if it were possible to include a battery meter inside the battery; but that’s another story.

  1. Sensor ID values. Yes; sensors.h is the starting point for when I start to integrate a new device; and of course there is the risk of treading on someone else’s ID. May I suggest having an accessible master list of ID values and a method for allocation of new ones instead of guessing. I just added mine at the bottom, and used the next number having counted all the rows in the array.

I have since added sensors.h data for ENS160 and ENS210.

ENS210 values might duplicate those of inbuilt Urban board sensors; I would like to see the data before deciding which set should be eliminated; or whether averaging should be applied. Thus I consider it essential that the platform should receive this data; at least for now.

  1. It’s unclear to me but did I edit the correct file ?
    I am aware I could not deal with the first part of the file; but from what I saw; most requirements were handled except for AQI calculations which I suspect are a new idea, calling for new work, but noting that ENS160 calculates it internally but only for VOC. There are several flavours of AQI of course.
    I know nothing of channels and how they work. So I’m presently leaving it up to you since I can only handle this if I have the complete picture.
    All of the new sensors require only ‘cleaning’ to remove aberrant outlier values (zeros, nulls, non numeric and extreme high values)
    Currently; the PM data requires aggregation into one set of readings; which is already present.

  2. Yes, of course a header file SCAS2100BP.json Is needed. It contains items such as those ID values that I do not understand; it’s best if you just create this file. It also contains a link to the location of the above file containing [sensors] and [metrics] data.

Concerning calculations.json file if I were to attempt to create such a file I would need to understand what are the variables used and how they go together in the actual calculations which would look something like those given in the Spec Sensors documentation. Of course those calculations are using formulae given by Alphasense but they ought to be compatible given they deal with the same mathematical problem based on the same technology.

I do not have such knowledge; I would be flying blind; it’s inefficient to pass this task to me unless you are going to spend the time necessary to fill my knowledge gap.
I previously sent you weeks ago a file containing data and connections for the spec Sensors devices. Is it enough for you to populate the calculations.json file.?

There remain open questions about the voltages I am seeing from the ADC right now. Unconnected inputs show readings of around 0.25 volts. Is it an artifice of the ADC and associated circuit : an offset voltage that applies to all inputs ? Or is it likely due to noise and could be ignored.? The point is whether it should be taken into account in calculations or not.
The Vref values should be 50% of Vcc but are not. It’s a 1 Meg ohm voltage divider connected to the input of an opAmp. I will try to measure those resistances and see what’s happening.

If I attach an oscilloscope to the wires concerned. Its input impedance (10 megohms) will affect measurement to a small degree. I can try this and see if it helps solve these questions.

Cheers

At this time my mind turns to OTHER things that could be causing the device to halt when a new config is entered.

I am flying blind on this thing because so am not completely familiar with all aspects of the code; notably the ESP.

But, it’s possible to make some educated guesses as to what could be happening.
For example; the history of this particular SCSV3: which began in the first half of 2021. It has been turned on spasmodically since then; collecting occasional readings; and with differing but increasing sensor count.
Recently; new sensors were added of which the platform has no knowledge.
I am unaware of how this is handled; but it seems like the platform must be preconfigured to handle the sensor ID’s that are presented in the readings. How does it handle sensor ID’s it does not know about ? Does it simply ignore the data? Seems like it.

However I observe that the SCS is telling me that it’s sending several sets of readings to the platform. It’s doing the same readings time and time again. Are the readings not being acknowledged and then deleted ?
Is it normal ?

When readings have not been sent (I think) they are stored in both eeprom and flash and SD card. Need to check this. But which of these storage locations are used as source for sending data to the platform.

My hypothesis is that (maybe) there are a bunch of old unsent banked up readings; and these might be choking the system.
I know for certain that the platform storage (for this kit) has a lot of isolated readings; not much good for anybody.

So I propose to take the following actions:

  1. Fix the platform config so that it’s aware of all the sensor readings that are useful to be recorded. This process is under way; @oscgonfer is helping albeit slowly due to a busy schedule. I have to wait.
    All that I can do whilst waiting is to clean up the firmware code removing debug and making small refinement’s.

There are no more firmware additions remaining to be done; but so I have in mind to move the I2C mux code into its own class for stylistic reasons only. ( It’s working fine now).

Only When platform config is done:
2. Reinitialise the kit on the platform and get a new token and identifier and link this to the new platform config right from the start.
3. Reinitialise the SCS storage locations using whatever tools are available. Delete files on the SD Card; empty eeprom if possible. Have the SCS rewrite it’s config like a new kit.
4. Bring the SCS Online with a fresh start and see what bugs it then exhibits.

It’s not such a bad thing to have to restart the SCS after entering a new config; it’s just that it normally it is not necessary. Until recently it was not necessary.

If this is the only issue it can be forgiven; but it’s essential that I discover any remaining OTHER bugs through longer testing periods. In particular identifying lock-up conditions; because that would mean the kit cannot be left unattended for long periods. The CO2 sensor is worrisome; still reading high values. I am think of ordering a replacement; pre-calibrated version.
The system must run continuously unattended for over 24 hours at a stretch because the SCS normally performs a reset in that time frame.

While the waiting at stage 1 and testing is going on; I can get the Wifi > 4G gateway going. Then it can be placed inside its enclosure and attached to the SCS.

Concerning the ADC Board and Gas Sesnors.

I made two sets of measurements using GW Instek MSO 2204AE Oscilloscope with 1x 200 MHz passive probe. Input impedance 1MOhm // 16 pF. <<. It’s low enough to load the high impedance signal.
I also used 2 different handheld digital meters.

  1. Resistance of Vref wire to Gnd and Vcc (using Digital Meters: Inconclusive. Was supposed to be 1 MegOhms in each case. Instead I saw 0 ohms to Gnd and ~ 6 MegOhms to Vcc.
    I fear this indicates the Gas Sensor Analog Interface boards are broken.

  2. Offset voltage seen on ADC Inputs that have no sensors connected. Unequivocally there is a DC voltage of 0.2 V and a noise signal ~40 mV on top. The noise signal is in fact seen on all of the ADC wires. I live near (<1km) a military/civil airport; so there is electrical noise from Radars etc ; In addition there are several electronic devices emitting radio signals in close proximity, including the SCS itself emitting 2.4 GHz wifi signal. I am surprised the measured signal is as low as it is.

Are there any conclusions. ?

  • It is a safe bet that there is > 50% probability that the Gas ASB Interface boards are broken. I must decide whether to also order the sensors themselves; it doubles the cost. On past performance; the replacements should be here in a week or so.

  • Can we carry on with work to support Gas Concentration voltages to ppm values ? Yes. The voltages are within the expected range.

  • Does that ADC Board contribute an Offset voltage? Looks like yes. But it is applied to both Vgas and Vref the same; and so I think that the offset should fall out in the equations; which simply compare the two voltages between each other.

I have an idea concerning the gas sensors.

There are two ADC Boards; each with two ADS1115 chips;; providing a total of 16 distinct single ended Measurement channels; of which we are using only 10 to connect gas sensors.

What if we connected Vcc to one spare input, and Gnd to the other; and took a double ended measurement; which should equal the voltage we know; being 3.3 volts.
If its not; thats an error in measurement; and the difference can be applied as a correction to each measurement channel.
This could be done before readings are sent to the platform; but it would mean it is unnecessary to apply any calculated or configured offset to measurement voltages; and it removes all doubt as to the veracity of readings.

What say you on this @oscgonfer ?

Concerning Sensors for this system and “too many readings”:

We need to decide how this is done, becase they are a lot of sensors and we need to decide which ones are to be sent and not, so anything from these lines will be ignored in the platform unless the sensors are there. I suggest going with Wind Speed, direction, one or two of rain and that’s it. The reason for this is the json package size.

OK: I have gone through the [sensors] block in the Blueprint file.
Firstly I have no idea what target I am chasing. How much is too much ? How many are OK ?
I can (now I have done so) reduce the number by a few, but in fact I actually need to see the data for some of the sensors to make a decision.

For example: CS811 measures eCO2 and tVOC; but the sensor appears to be dysfunctional; slow, and others have complained about it on this forum.
ENS160 represents one of the may alternatives. I found it when enquiring with ScoSense about which Gases are measured by CCS811. They cannot tell me (they bought the product from someone else, but they offered me the ENS160/EBS210 to try out as an alternative. And thats what I intend to do. Basically put the readings from CCS811 and ENS160 alongside one another; see if they correlate. If they do, then ditch the CCS811 because its unreliable to start up. ENS160 I have already discovered was easy to implement and worked the first time I tried it. That means its more likely CCS811 will go than the ENS160. If you press me then I will eliminate CCS811, not ENS160.

Concerning Rain: Yes: the device puts out several Rain Interval Readings; of which I think that only rain interval (immediate time window) and Rain Per Interval (rain rate over the last hour) are useful. So, I have eliminated the 3 other measures for now. They will still be collected, just not sent to the platform.

Concerning INA: The reason for including it in this system is because the system will run on battery (it is now); and the amount of current drawn determines battery life, its an easy calculation. Its easier for me to display on the platform; but I can get the amount of current drain by checking using the UI.

Concerning PM readings. There are two devices generating 3 readings each that are sent to the host and there I am not certain what happens. What I think should happen is that the two sets should be averaged. What I think is happening is that the ‘min’ value is being picked:

 "EXT_PM_1_CLEAN": {
      "desc": "PM1 calculated based on both PMS5003 PM1 inputs",
      "kwargs": {
        "factor": 0.3,
        "limits": [
          0,
          1000
        ],
        "names": [
          "EXT_PM_A_1",
          "EXT_PM_B_1"
        ],
        "pick": "min",
        "window_size": 5,
        "window_type": null
      },
      "process": "merge_ts",
      "units": "ug/m3",
      "post": true,
      "id": 89
    },

If so then I think it needs to change. Not sure how to accomplish it; but it is possible to do this on the device itself fairly easily; then send only one set of readings through; eliminating a further 3 readings. If done on the device; then only ‘clean’ post processing is needed.

Concerning Gas Sensor Readings:
I have made some changes to the blueprint file and included a linkage from the [metrics] section to the [sensors section] like this:

"CO_WE": {
      "desc": "CO working electrode raw value",
      "id": "X001",
      "kwargs": {
        "channel": "ADC_4A_0"
      },
      "post": false,
      "process": "channel_names",
      "units": "V"
    },

(I added the ‘channel’ reference’)
I am uncertain what other parameters need to change; as I do not know how this is used in code (the logic).

Thereis another related piece of this puzzle:

"O3": {
      "desc": "Calculation of O3 based on AAN 803-04",
      "id": 157,
      "kwargs": {
        "ae": null,
        "alphasense_id": null,
        "from_date": null,
        "timezone": null,
        "t": "EC_SENSOR_TEMP",
        "to_date": null,
        "we": null
      },
      "post": true,
      "process": "alphasense_803_04",
      "units": "ppb"
    },

It refers to EC_SENSOR_TEMP; which is a back reference to Alphasense PT1000 device; not fitted to this system. Instead we intend to use PM_DALLAS_TEMP. Since this device reads temperature directly, I am uncertain if it can be entered directly to replace the EC_SENSOR_TEMP reference or whether there is another step.
Note: If some issue arises from using the Dallas Probe then there is in fact a temperature sensor on each Spec Sensors ASB, it would need to be wired via a separate ADC channel and indeed, voltage readings would need conversion to deg C in that case. That wiring would be messy. But lets not get ahead of ourselves, we are going with Dallas unless it proves problematic. It should be more accurate than any thermistor.

Finally, concerning calibrations file; having looked at it; as aforesaid; I would not dream of entering parameters into this file without understanding where they plug into the formulae; how they are used. Send me the formulae and I can look at it; otherwise its your job.

As it stands; we have the following data:
(a) Sensitivity Code n nA/ppm
(b) TIA Gain in Kv/A
(c) Calibration Factor in V/ppm
(c) Vgas
(d) Vref
(e) Temperature.
(f) Its also possible to compute Cross sensitivity. It is provided by Spec Sensors in their data sheets. Examples;
SO2 Sensor is also sensitive to H2S (equally sensitive). Luckily we measure both, and can compensate.
Ozone sensor is equally sensitive to Nitrogen Dioxide, but Hydrogen Sulphide reduces O3 Reading.
NO2 sensor has no cross sensitivity
H2S sensor has 1 in 25% (slight) cross sensitivity to each of SO2 and CO
CO sensor has no cross sensitivity

Effects if not x-sensitivity is not compensated.
If H2S is present; then I would expect that SO2 sensor would also light up; but O3 sensor would darken.
If NO2 is present then O3 sensor will light up too.

So, cross sensitivity is going to need a kind of matrix calculation.

{ I have not included mention of gases we do not measure therefore unable to compensate for}

I found a couple of mistakes in the ‘specification’ document I assembled for Spec Sesnors; so I will correct those and add in the Cross Sensitivity data too.

To help things along; I will also assemble a lookup table for temperature compensation for each sensor; which involves taking data off the graphs. For Taiwan; we need only to cover temperature range from -5 degC to 45 degC.

Here is a summary of recent activity concerning the ADC Board and Spec Sensors Gas Sensors.

I saw some anomalous readings output from the SCS UI interface.
There was concern that the Analog Sensor Boards had been smoked by accidental application of 5 Volts Vcc instead of specified 3.3 V supply; and the anomalous readings derived from that incident.

There are two outputs from each gas sensor that are being measured.
Vref is a reference voltage that is approx 50% of the supply voltage.
Vgas is the voltage that is converted from current output of the Gas Sensor and amplified.

The Vcc supply is around 3.3 V; therefore Vref expected to be ~1.5 V but it was over 2 volts.
I got some help from the manufacturers Engineer who supplied me the circuit diagram.

I conducted measurements using a Digital Voltmeter and Oscilloscope.
One fault was found to be that Vref is connected to Pin A0 of the ADC instead of Pin A1 expected. The source of this error is irrelevant; but it affects calculation of results and configuration of Platform json files.
The voltage on Pin A0 is within the range expected for Vref on all of the sensors; making some allowance for different bias voltages one each type of ASB (3 types).

Measurement also showed unexpected frequent variation in Vgas signals. Gas Sensors readings of course should vary with exposure to the target gas; but this is not something that occurs quickly in normal settings.
The testing showed that a spurious signal of approx 50 Hz plus noise at a level 1 - 2 Volts is periodically seen on the Vgas sensor connection to ADC. It is a burst of signal lasting a second or so, followed by silent intervals.
I searched for similar signals within the Smart Citizen system.:

  • Voltage Supply ? : In fact the SCS was supplied from a battery during the testing period. No mains supply to provide a source of ‘ripple’ sometimes seen on unregulated power supplies.
  • Serial Communications lines ? : There are 5 serial comms lines in SCS (2 x PM Sensors; Dallas Probe; Wind and Rain Sensors). None of these were operating during the testing.
  • I2C Bus ? The I2C bus showed occasional activity; but interfering signal from I2C was not seen on the Vgas lead and its timing did not coincide with the pulses of spurious signal.

So, I am forced to speculate that the spurious signal originates from remaining possibilities:
(a) A ‘parasitic’ oscillation originating from the ASB Board itself. This seems unlikely; mainly because the oscillation occurs in pulses; parasitic oscillation would likely be a continuous signal.
(b) External, nearby source of EMI. This seems most likely; because the location of my ‘lab’ is nearby an airport that is used by both military and civil air traffic. They have Radar installations that ‘sweep’ by rotating an antenna. This is a good match to the characteristics of the observed spurious signal.

The question is what to do about it. It is clearly interfering with the measurements.
If the signal originated from a wide-band radar signal; then its possible the signal level could be reduced by installation of “Ferrite Bead” type of inductive devices. The inductive impedance increases with frequency. These are installed on signal wires and effectively provide a low pass filter in conjunction with any existing decoupling capacitors and RC elements.

The idea is to install the ferrite beads on individual wires found to be susceptible to interference either source or sink.
A disadvantage of using standard ferrite beads is that they are most useful at high radio frequencies; 100 MHz and above. I do not know the centre frequency of the interfering signal (I think the 50 Hz is a sub-harmonic). Published information about airport radars indicates frequencies between 1 GHz and 6 GHz are often used; But I know from life experience that an audible signal can often be heard on AM radios much lower than that. A 1 Ghz and above Radar signal should be effectively blocked by a ferrite inductor.

Other measures that can be taken are: ‘average’ the Vgas readings over several samples. The interfering signal is superimposed on the Vgas DC signal; thus both increasing and decreasing measurements; depending on the time-instant and time-window-duration of signal sampling by the ADC. One might hope that averaging the signal readings over time will pick up both positively influenced and negatively influenced samples in even amounts. But it’s a game of chance and the effects likely random. The more samples that can be averaged; the more accurate will be the result; reflecting mostly the average gas conc. level.

I will discuss this with the smart citizen team to decide on the best approach (including whats actually feasible in the platform).
@pral2a1 @oscgonfer @victor

I also ask myself WHY is this affecting the Vgas signal and not other wires/signals such as Vref.
Well I think that it is possible the signal is being pickup by the gas sensor plates; perhaps there is a resonance with the physical dimensions of the sensor plate at the frequency used by the Radar; they act like a dipole antenna. The small signal then amplified by 500 in the op-amps forming the ASB buffer amp.
Bear in mind that Airport Radar operates at KiloWatt power levels: very strong signal nearby.

I will order some ferrite beads from Digikey; purchase from local Taiwan electronics shops is not possible right now due to the lengthy public holidays of Chinese New Year. Luckily they are fairly cheap to buy.

I have moved away from the idea of using ferrites by themselves on the Vgas leads.

Main reason is I took a look at circuit diagrams for both ADC and ASB boards. That darned Vgas wire is just a wire that runs from output of an op-amp in the ASB to the input of the ADS1115 which is high impedance. No passive components at all; no resistors, no capacitors.

If I just add the ferrite it effectively adds series inductance to the wire which will increase its “resistance” with frequency; but that resistance starts at a low value. It can work like a low pass filter but only with existing passive components providing a pathway to Gnd for the spurious AC signal and there are none. I might get some of the attenuation I need but not enough probably.

I looked for EMI filters and the like but found none suitable for filtering out the low frequency component of the spurious signal.

So, now I am looking at adding passive RC components to the connector plug : something like 330 uF and 100 ohms series resistor will attenuate signals from ~30 Hz and upward to get attenuation of something like 20 dB @ 50 Hz. It should work and I can breadboard it to confirm.

Does anyone have other suggestions ?

The spurious waveform has a period of ~25 mS per cycle = 41 Hz and the period of the burst is 1.25 Sec long with gaps in between (approx) 2 Sec.

The peaks of the burst are phase modulated with a repeating pattern (looks like data). This shows in the time domain zoomed out view, but appears like jitter when zoomed in …

The amplitude of the signal is ~160 mV (varies).

The amplitude of the signal is the same for 3 gas sensors; but the superimposed data modulation mentioned above has a different pattern.
Here is an image of the waveform to allow readers to visualise it.
The image shows waveforms emanating from two gas sensors Vgas leads.

OK… So Today was the end of Chinese New Year holiday. My parcels from Digikey arrived and I was able to visit the local Electronics shop. My new Spec Sensors Gas sensors also arrived.

  1. Fabrication of RC Low pass filter. There are a lot of steps involved that I will not describe here, but its sufficient to say that I arrived at suitable values of 820 ohms and 22 uF that achieved a ~20 dB reduction in the level of the spurious 41 Hz signal previously described. It is enough of a reduction to eliminate this source of variations in gas sensor readings. In fact the filter also attenuates much higher frequencies up to 10 MHz. In addition the effect on DC voltage measured at the input to the ADS1115 was also about 0 volts. This was based on measurements made using signal generator in the Oscilloscope; and also the ‘Frequency Response’ software function (which plots 20 Hz - 25 KHz Bode Plots).

Thus the filter met its design goals and so moving on to step 2:

  1. I made a temporary installation of the filter in line with the H2S gas sensor (picked at random); and verified that the spurious signal was still present; and that the filter attenuated the real signal.

  2. Because it was necessary to disassemble the gas sensor -+ADC + OLED board assembly which is tedious; I took the opportunity to open the ‘new’ H2S sensor and check it out, comparing with the old one and see it reflects damage or not. I installed it in place of the original that had been used for testing earlier. It looks like the old one is not damaged. At least the voltage level of Vgas is the same on the new sensor as the old sensor. The old sensor was reinstalled.

This test was done with only the ADC Board and the one H2S sensor connected; all other gas sensors and devices on the I2C I2C Mux distribution were disconnected (due to the disassembly).
When I reconnected the Oscilloscope; I was surprised to find that the spurious signal had disappeared.
you know what I was thinking at this point ! But I suppressed emotions and carried on methodically…

So, I proceeded to reconnect items one by one to see if the object emitting the spurious signal would show itself.

Only when I reconnected the NEOM8U GPS signal and waited for the Data Board firmware to initiate GPS did the 41 Hz signal reappear.

Now, this is awkward. Firstly I do not have a replacement for it. Secondly this device is used to augment the 3.3 Volt power supply supplied by the Data Board; which does not have enough capacity to run the entire system by itself. Without the help from the GPS board the devices on the I2C bus croak when the OLED is connected. There is risk of damage to the Data Board power 5V/3.3V translator circuit when it is run for a long time overloaded.
It’s also a bit of a mystery: The GPS is a Radio Receiver at very high frequencies; transmission of a low frequency 41 Hz signal is entirely unexpected.
I have to do some research as to what is the origin and purpose of this signal; see if it can/should be disabled; or if somehow it indicates a fault in the device.

If I can eliminate this signal; does this negate the need for the Low pass filter (??): I do not think so. Using the oscilloscope I still see a number of noise and mains ripple on the Vgas lead from the sensor. I think the sensor plate acts like an antenna picking up any noise that happens to be around the place. This noise can be at any level; and the high amount of gain in the Analog Sensor Board(s) means the noise is amplified, and may then interfere with actual readings which are DC.
Concerning the LPF: I now need to work out the best way to install the 0.25 watt resistor and Electrolytic capacitor at either end of the little cables joining ASB’s to ADC’s. They were neat and tidy; now they will not be so.

@oscgonfer @victor

Today I find myself at an impasse. Trying to work out which task is the logical next step.

  1. The source of pulse 41Hz spurious signal has been pinpointed to the Neo8mu GPS. I have reached out to the manufacturer support to ask them to tell me if and how it can be turned off in firmware. So far one response indicating they cannot properly read and understand English.
  • The issue here is that it’s easy to solve the noise problem by simply eliminating the gps from the design. But it’s not as easy as it seems because:
    a. GPS was included to have geographic location of readings recorded inline with sensor data. I am loath to remove it and dispense with this essential data.
    b. There is not enough 3.3 v power to run all of the devices in the system without it. In particular; without the gps there is not enough PoweR to run the OLED display.

  • So, to break that impasse I have ordered some Sparkfun Buck-Boost DC-Dc converter boards that can supply 3.3 v at 2 amps: enough to supply the whole system. The plan is to connect this supply to the system via the Grove I2C wiring. I just plug it in to spare grove sockets carefully selected to provide minimum voltage drop to all the other devices; likely connect at the I2C Mux.

  1. Other tasks that need doing need to be prioritised:
  • install the LPF(s) onto each gas sensor cable in some neat arrangement that doesn’t look too much like the hack that it is.

  • Install and test the new power board

  • experiment a little with NEO. GPS class code and try out likely library function candidates to disable that time signal causing the issue. Not wanting to spend a lot of time on this without some signal from Neo support. If you look at the library you will see why. There are literally hundreds of functions that are poorly documented and none explicitly named for the signal that appears to be causing the problem.

  • continue to patiently wait for Fablabs to do their end of the platform integration and display of all the sensor readings. When it’s ready: test and verify. As far as I am aware I have provided all the info that I am able to.

  • reassemble the SCS and continue testing and debug the firmware.

  • get the Wifi- 4G gateway working; then install into its enclosure and attach it to the main SCS enclosure.

  1. I have to say that this continuing series of impediments is taking its toll on my enthusiasm. I had hoped to have this system ready for deployment 9 months ago ! Whilst I have learned a lot I have to say it looked a lot easier going into the project than it has turned out to be. I just hope that my experience can be helpful to others and certainly the added sensors and capabilities of SCS should prove to be useful as time goes by.

SAMD51 boards of any variety are the proverbial unobtainium right now; no doubt due to worldwide chip shortages.

So when I saw that it’s possible to purchase the ‘Metro’ version of Adafruit’s SAMD51 development board today I added it to my order.

It will be some time before I find the time to jump into it; however.
The Metro version has fewer resources than the upmarket ‘Grand Central’ version; but there is still waaay more Ram and Flash than found on the SAMD21G.

meanwhile the shortage of SAMD21G chips continues as before; no doubt the reason Seeed have run out of SCSV 2.1 kits.

If anyone knows someone that has a couple of spare “bare bones” kits around, I have need of one (Data board + Urban board only) that are tested and working.

Concerning the issue with 41 Hz pulses seen at the interface between Gas sensor ASB and
I reached out to Ublox support and also on their public discussion board. I received a lot of responses.
The summary:

  1. The signal is NOT a normal signal emitted by the Neo M8U gps.
  2. The most likely cause is power supply instability augmented by overload conditions.
  3. The issue concerns the 5v supply rail and is not seen on the 3.3 v rail.
  4. The ADC board that connects the ASB gas sensor boards is supplied only by 3.3 Volts obtained through the pathway Grove I2C connections ← I2C mux ← gps in parallel with data board. That pathway incurs a voltage drop of around 0.05 volts.

So; something still does not gel. (3) does not agree with (4). Upon the available evidence it seems that in fact ripple is present on the 3.3 v rail at a very low level. Another possibility is parasitic oscillation possibly caused by track to track emi.

The gas sensor ASB Board takes the 3.25 volt supply through a resistive high impedance divide by 2 voltage divider to provide ~1.57 volts.This forms one input of an op-amp; the other input being connected to the Gas sensor plate. The Op-amp has a high gain (between 50 and 500 depending on the gas sensor chemistry ).
A ripple signal at low level is seen on the gas sensor output continuously but this level increases dramatically when pulses are seen.

So, we need to determine the source of the continuous ripple as well as discover why it pulses.

The current hypothesis is that the power supply is overloaded and/or unstable and/or weak design and this is the cause of the ripple. When additional load is placed on the power supply eg. when gps operates the amount of ripple increases.
The previous hypothesis was that the ripple and pulses were induced onto the gas sensor plates by EMI emanating from Nearby radar. The problem is that no radar uses such a low frequency. The possibility of the EMI emanating from a source within the SCS was not seriously considered. The source of this signal to me remains conjectural. This hypothesis is supported by the fact that alteration of gps timing parameters does not alter the signal characteristics.

The proposed solution is to replace the two main 3.3 volt power supplies (data board and gps board) with one higher capacity unit. The proposed replacement is a buck-boost power supply with input of ~5 volt and output of 3.3 volts and capacity of 2 amps. It is intended to adjust the output voltage to take voltage drops into account delivering a value close to optimum 3.3 volts at the points of use.

The supply distribution is planned to utilise Grove I2C wiring: the PSU will be connected at the centre of the star bus topology network formed by connections to the I2C mux unit.

Potential issues/risks revolve around the type and dimensions of circuit board tracks in the central component eg. the I2C mux board. However this risk already exists with existing power distribution to sensors from data MCU board and gps board.
A further risk arises from the design of the selected buck-boost psu.
(a) the ac voltage level and frequency of ripple at the output.
(b) whether the amount of ripple increases when current draw changes rapidly eg when GPS activates.
(c) how well the psu and sensor mounted decoupling capacitors work to reduce voltage effects of logic switching reflected onto the supply rail.

Backup plan
If necessary, a second buck-boost psu could be included to provide better isolation between high and low demand sensor users.

OK: Closing the thread concerning spurious 41 Hz signal appearing at the interface between Spec Sensors ASB and ADS1115:

Disconnection of the 5V supply going to Sparkfun NEO M8U GPS and connection of a Buck-Boost 3.3 V 2A supply via Grove I2X Mux board has removed the spurious signal entirely. 5V supply to the Data Board is only connected when the system is connected to computer via USB; which is unavoidable.

What I observe instead:
The Buck Boost 3.3V output includes a 50 mV non-sinusoidal (complex waveform) ripple component @ 2.4 MHz. This ripple component is seen at the ASB output point; amplified to a level of 1V P-P. The ripple signal is adequately attenuated (-18 dB) by the RC filter previously mentioned in this thread. I have not yet decided whether the RC filter should form a permanent part of the system. I will discuss with Spec Sensors; because I have formed the view that the ASB ought NOT to be amplifying every stray bit of EMI thats around the place. The design of the ASB actually includes a regulator chip that has been left out of production boards.

Whilst the main 42Hz pulses signal that initially aroused attention has been removed; noise from the buck boost power supply is still high enough to affect readings from the gas sensors. It’s a 2.4 MHz signal 50 mV. at source; 1v p-p at Vgas output point due to amplification in the ASB.

The conclusion of issues around the Gas Sensor boards would best involve making the ASB less sensitive to noise/ripple on the power ie an internal LPF in the feedback loop of the internal op-amps. After all the signal of interest is purely DC. (0 Hz)

I am not about to modify them myself so I have asked the manufacturer for a remedy. Waiting to see their response.

Failing a positive response from them; I am also trying to devise a comfortable way to deploy a LPF at the interface.
Yes, I have already designed a RC circuit to dampen the Vgas signal; but it makes more sense to reduce the input Vcc ripple signal. So I have devised an LC filter that can be applied to the power input: 1mH + 100 uF ought to do the trick. It has a cutoff frequency (-3 dB) around 500 Hz.
I might even deploy both: input LC and output RC.
testing will help make the decision.

The main issue is physically mounting any kind of filter. The space is very confined.

OK: Today’s report.

in the past week or two I have:
(a) deployed a different power supply of type buck-boost which provides 3.3 volts @ 2 amps capacity. It is connected to the rest of the system via a Grove Cable plugged into a spare port on the I2C Mux device. The power leads on this device connect directly to each connected sensor and are not switched by the mux. The characteristics of this power supply include ~50 mV p-p of 2.4 MHz ripple. On Test connection, the output voltage does not change in response to periodic higher power draw by the GPS device.

(b) the 2.4 MHz signal was found to be amplified by the highly sensitive op-amp circuitry found on the Spec Sensors analog interface board. Thus the 50 mV signal turned into 500 mV at the point where its measured by the ADC board. This is undesirable; therefore a 3rd Order RLC low pass filter was designed and deployed at the point where Analog Sensor Board connects to ADC Board. It is not completely satisfactory solution but it achieves the goal to silence the ripple signal. Spec Sensors suggested to use a 3 V coin cell power supply for the Analog Sensor Board; however this has the disadvantage that the battery will run flat within 2 - 4 years depending on whether CR2032 or CR3032 battery is used. The larger CR3032 batteries and holders are not easy to get ahold of either.

(3) So, Now I have reassembled the SCS V3 and carry on with debugging of the firmware.
There is a known issue with the Noise Sensor and I2S communication which is very difficult to solve and so the Noise Sensor is disabled at this point; allowing other issues to be resolved without that issue getting in the way.
I also have the OLED display unplugged from the system. There is nothing wrong with it, but cable was in the way when I was working on the gas sensors.

The operating characteristics of the system under these conditions:

  1. When the system is started all of the connected devices are ‘found’ and activated. This indicates that the I2C Mux is operating as expected.

  2. In overview, after new firmware is loaded; the system starts normally. Only after I enter config details does it begin to misbehave.
    2.1 If I load the config in ‘network’ mode the system starts to take readings from sensors but then seems to halt; and the serial UI becomes unresponsive. It appears to operate normally for one or two complete cycles before it halts.
    2.2 If I load the config in ‘sdcard’ mode then it operates for longer (~3 minutes) but then halts.

  3. In both cases when it halts is does so at different, apparently random points in the cycle. From this I conclude that something is happening that is likely unrelated to taking of readings and saving them or transmitting them.

  4. I would dearly like to have a version of firmware that can include debugging symbols; allowing me to utilise the Atmel ICE debugger; and it should lead me to the place(s) in the code that are problematic. But it’s not the case. I am forced to use other less efficient techniques.
    The problem making firmware with debugging symbols included is that this makes the size of the compiled firmware too large to fit into the available memory of the SAMD21 device. This is the main reason I am a proponent of moving the system onto SAMD51 which has twice the memory of SAMD21. But again its not an available option today. [ I do have a Adafruit Metro M4 board available however. ]
    The issue with Atmel ICE is not limited to getting of a debug firmware. The Atmel Microchip Studio software tool operates only under Windows. I have Windows 11 running under ‘Parallels’ environment; and its a 64 bit environment. In this environment the Microchip Studion must access ICE device using USB connection; but I have so far been unable to get the device to be recognised. I can use platformIO but aforementioned size issues prevents that.

  5. What I intend to do today is to try disabling in turn (a) Urban Board and (b) Aux Bus and see if the operating characteristics change for better or worse. Urban Board is disabled by simply unplugging it. Aux Bus is disabled by unplugging the Cable.

wish me luck…

Well, That did not take as long as I thought it might.

I started by disabling the GPS in firmware (by edit of the sensors.h file).
I ran the system for a while in sdcard mode; and it appeared stable.
Then I enabled network mode; and it also appears stable.

So heres the thing with the GPS.
Its integrated into SCS firmware I think mainly to satisfy the needs of the bicycle project; and for that use case it needed to update the position and speed of the system very often. it takes position fixes as often as 5 times per second ; and uses quite a lot of system resources to do so. In particular I2C Aux bus resources are hit pretty hard; one of the reasons I had so much trouble getting the I2C mux working stably a week or two ago.

Now, in the case of my project; GPS is provided to fix the location of a set of sensor readings; but the system as a whole does not change position very often at all. Like monthly maybe. I can be satisfied with GPS readings of just longitude, latitude and altitude taken once daily. Additionally, this system contains more sensors operating on Aux bus; each of which needs to be read up to once every 5 seconds.
So its a question of how much I2C Aux bus resources are used and for what purpose. My system needs to lean towards providing the resources for all of the sensors and not giving too much to the GPS.

Given that frequent reading of the GPS is contributing to system instability I will discuss with fablabs team (@pral2a1 ) what changes can be made in firmware to accommodate both my requirements (once daily readings) and those of Bicycle project (very frequent). I am thinking it needs a config item to change behaviour of GPS readings.

UNder condition that:

  • Noise is disabled

  • OLED is unplugged

  • GPS is disabled

  • Network mode enabled; SCS V3 is publishing data to the platform; albeit the platform profile for kit 14812 does not match the actual profile. Gas readings are not being calculated from ADC voltage readings. CO2 direct reading in PPM does not appear; Secondary Trial ENS160 VOC sensor including AQI is not visible. Evidence is that the profile json file that I prepared is not actually being used.

Finding: After starting the system at about 15:00 yesterday; I find it is still running today at 10:00 am. I think it is reasonable to say that it is stable.

So now, the task is to re-enable the 3 missing components one by one to confirm which ones destabilise the system. Previous findings indicate that:

  • NOISE will cause a lockup; and the root cause is an issue deep within the toolchain that is used by the compiler to build the firmware.
  • OLED MIGHT cause a lockup; and it is likely because the OLED display is updated frequently; and the likely cause is overload of the I2C bus.
  • GPS DOES cause a lockup and the cause is unclear. The lockup condition appears at random times after a reset. Likely also due to overload of the I2C bus. It should be deterministic but I have not tracked down the root cause yet.

Again… wish me luck. The goal is to try to get them all working. Priority order of importance is GPS > Noise > OLED; but I will tackle them in order of (1) Noise (2) OLED (3) GPS

OK. (Attention @oscgonfer )

First step completed.

Noise Sensor was enabled (in sensors.h) and new firmware installed.

This version has i2s.begin() in noise:start with no matching i2s.close() statement.
{ I2.close is located in Noise:stop() }

Results: System locks up after 2 sets of readings are taken. The last statement before lockup is:

11:09:52.795 → (2022-02-23T03:09:49Z) 30 readings saved to sdcard.
11:09:52.937 → Connecting to Wifi…
11:09:53.166 → ESP finished booting
11:09:53.271 → Connecting to Wifi…
11:09:57.588 → Connected to wifi!!
11:09:58.626 → Network publish OK!! (30 readings)
11:10:02.048 → Last message repeated 10 times
11:10:05.467 → Last message repeated 10 times
11:10:08.915 → Last message repeated 10 times
11:10:12.323 → Last message repeated 10 times
11:10:15.753 → Last message repeated 10 times
11:10:19.192 → Last message repeated 10 times
11:10:22.616 → Last message repeated 10 times
11:10:26.072 → Last message repeated 10 times
11:10:29.482 → Last message repeated 10 times
11:10:32.921 → Last message repeated 10 times
11:10:36.356 → Last message repeated 10 times

(which isn’t helpful I know).

Without adding some debug I do not know exactly where in the code its getting lost. Forced into conjecture I would say that it seems likely that this is the old hoary DMA issue rearing its head again. And because you had this (Noise) code working previously we might extend the conjecture to suggest that the difference is that I am using Aux Bus somewhat more extensively than is found in your stock standard SCK2.1; and the commonality is that both I2C and I2S are using SERCOM and associated with DMA. So if I were to pursue this issue thats where I would go looking.
But I do not intend to pursue this further. It is beyond my brief; and beyond my resources and skillset.

I can simply say that the NOISE sensor deployed on URBAN board appears incompatible with the way in which I am using the SCS system; and my only pathway to progress for the overall system is to disable it. It’s a pity; because noise in the urban environment is surely a key pollutant affecting liveability; and I would have liked to include it within my comprehensive SCS V3. So its disappointing. :frowning:

As a final check: I reverted the code within NOISE (part of sckurban) to the distribution version and rebuilt the firmware; installed it.
Upon Reboot; and after entering network mode, I find that indeed the system locks up like before. It only gets in one set of readings before it locks up.

So it is no better. My Executive Decision to disable the Noise sensor is confirmed.

After reverting the code and again disabling the noise sensor in sensors.h; I have confirmed that the system now operates as it should; without lockup.
Its time to move on to the OLED.