Back to Smartcitizen.me

Building the firmware

OK; sometimes things can progress quickly.

0/. Issue with lockup caused by Noise: I2S: Accidentally appear to have fixed this; Not sure if the fix is sustainable; time will tell.
1/. Rationalised reset of Mux at top of AuxBoards:getReading :: Done
2/. SCD4x CO2: I removed atmospheric Pressure compansation call. If that does not do the trick; then I will try a reset to factory config of he sensor; see if it comes to its senses !!

I have also removed pretty much all of the debug output that I consider superfluous. so the UI output is back to sane levels now.

What I see happening:
(a) I see ‘Light’ reading being taken 3 times. Not sure why.
(b) CO2 readings remain zero. I will wait to see if this fixes itself over a longer time frame
(c) I see this:

10:53:35.819 -> (2022-01-21T02:53:33Z) Readings saved to flash memory.
10:53:37.336 -> (2022-01-21T02:53:33Z) 51 readings saved to sdcard.
10:53:40.692 -> Last message repeated 10 times
10:53:44.226 -> Last message repeated 10 times
10:53:47.741 -> Last message repeated 10 times

Its like the host system is not acknowledging reception of readings that I am sending.

When I look at the smartcitizen.me map: readings display; there are anomalies. For example I can no longer scroll right to see readings that do not fit the display. The system seems to think I might have a UV sensor.

I think there is a possibility that the host is not understanding the readings that I am sending. It is also possible I have screwed things up by sending spurious data whilst the system has been undergoing debug.

@oscgonfer I am wondering if you might take a look at the host data for system xrw2e2 (Serial ID 8A76) and ‘clean it up;’ look at the config file: Please tell me what actions I need to take; perhaps amend ID values.
I would like to see the data that I am sending through…

OK; After having written that I had the thought that I should have a look at the docs and see what if any things I can do by myself to facilitate this process.

Thus I have started wading my way through the information given in ‘Handling Calibration Data’ .

I have no intention at this stage of trying to analyse data collected on my SCS locally on my own Laptop.

My goal at this point is simply to select the correct skeleton (if necessary create my own custom skeleton) and related files so that the central system knows what sensors I have installed; and can correctly interpret data that is uploaded.

At the moment thats not the case. My system is spewing out 51 different sensor readings and most of them are inaccessible via the Map.
Downloaded CSV file also does not contain all of the reading values; for example INA219 and SCD4x readings are not there. I have not checked all of them.

Whilst I have been reading through the material learning how to create JSON for device setup on the platform; the system has been going through the first stage of testing.

It has failed that stage. Looks like Mux lockup (again). But after an hour or so. So I have to investigate further and put aside the platform config for a while.

The SCD41 CO2 sensor is finally giving up some reading values. like 15281 ppm.
If CO2 levels were that high I would be very uncomfortable … gasping likely. I opened the window and it dropped by a couple of thousand PPM; but I still do not believe the CO2 level is so high.
Unfortunately its not so easy to get published data on this parameter; I guess its politically sensitive.
All that anyone ever quotes are relative values, not absolute ppm values (the reason I wanted to have this sensor working). But I expected it to tell me levels like 500 ppm…

I cannot put aside the feeling that this thing is fighting me all the way…

OK,

Moving on…
So Today after Sunday break; I tackle some of the issues I observe.

One issue is this:

09:57:01.259 -> Returned from taking a reading from Light
09:57:01.259 -> Light: 7 Lux
09:57:01.259 -> Base::updateSensors :: Reading sensor Noise dBA
09:57:01.259 -> Base::getReading Noise dBA
09:57:01.259 -> Reading requested from Noise dBA
09:57:01.259 -> sckNoise:getReading
09:57:01.259 -> sckNoise:getReading priming of I2S port
09:57:01.517 -> sckNoise:getReading finished priming of I2S port
09:57:01.562 -> sckNoise:getReading finished priming of I2S port
09:57:01.562 -> sckNoise:getReading finished priming of I2S port
09:57:01.562 -> Sck_Noise:getReading: All done
09:57:01.562 -> Returned from taking a reading from Noise dBA
09:57:01.562 -> Noise dBA: 35.94 dBA

This output is produced by this code (just a snippet):

uint32_t startPoint = millis();

	base->sckOut("sckNoise:getReading",PRIO_MED,true);

	//if (!I2S.begin(I2S_PHILIPS_MODE, sampleRate, 32)) return false; // moved to ::start

	// Wait 263000 I2s cycles or 85 ms at 441000 hz
	base->sckOut("sckNoise:getReading priming of I2S port",PRIO_MED,true);		
	
	while (millis() - startPoint < 250) {
			I2S.read();
	}
	
	base->sckOut("sckNoise:getReading finished priming of I2S port",PRIO_MED,true); 

Do You see 3 instances of the last line of the code ? No, neither do I.
Do you see anything like ‘GoTo’ in the code; that might repeat that last line of code ? Neither do I.

I cannot; for the life of me understand how or why the system would execute that last line of code 3 times without also executing preceding lines of code 3 times. This is beyond my experience; Everything I have ever learned about coding tells me this stuff is pretty much deterministic.

This tells me something is going on that I do not understand.

I will play with it a little and try to see whats going on.

OK:

I had previously disabled a mechanism in the SckBase::sckOut function that stopped repetitions of the same debug appearing in the output. <-primary cause of this issue

However as to the reason why this particular debug gets repeated remains beyond me. It has obviously been a concern to previous developers (eg fablabsbcn team) otherwise why put in the mechanism in sckOut.

In any case its not worth spending any more of my time on it.

Moving on to other more pressing issues

I determined to see whats happening with my 2nd OLED display.

I had put in some code that should have ensured that 2nd OLed display mirrored the first. But the 2nd OLED remains blank.

I connected the 2nd OLED to a seeeduino V4 (arduino clone) and compliled the example “hello seed” example.
The display remained blank. The example code uses u8g2 library; and the code does not tell the library about the I2C address. But there is a function that does so.
So I included that function: u8g2.setI2CAddress(0x3d);

The display remained blank.

I connected up OLED 1 (set to address 0x3c) and recompiled the code (changing the abovesetI2CAddress) appropriately; and this first OLED worked just fine.

I recall testing this previously; and it worked at that time.

So, I have formed the interim conclusion that perhaps the 2nd OLED has gone faulty.

I have to work out if its worthwhile to order a replacement from Seeed or Mouser or DigiKey. I am leaning in the direction of NOT replacing it;

Moving on to the next issue: why does the system hang immediately after I enter new config details (for wifi connection). When I reset the system after that, it works.
The system did not previously exhibit this behaviour; so something I have done has changed its behaviour.

OK:

Moving on to the next issue : why does the system hang immediately after I enter new config details (for wifi connection). When I reset the system after that, it works.

So, I have put in quite a lot of debug to trace the program flow after entering in new config details.
I have determined that in fact lockup occurs EVERY TIME config command is executed.
I have traced program flow through to the point where the config is saved into eeprom; and ESP is rebooted to pick up the new config details.
At this point program flow stops: specifically after returning from the ‘In()’ function.

I am unsure exactly what is supposed to happen at this point; but from what I have previously observed; the change of state caused by execution of config command causes the system to start reading sensors; compiling the readings and sending them to the platform.

What I think MIGHT BE happening is that because there is a mismatch between what my SCS system is sending and what the platform thinks I have got; this transfer has become problematic; and something is breaking.

It is clear that I need to pay some attention to the platform json data; and I will need some help from the @pral2a1 team to do this, perhaps @oscgonfer perhaps @victor

I will again study the materials provided to help me understand what is going on.
From what I have read to date; there is a json file that describes the skeleton of my system; and its the first jigsaw puzzle piece that would join my SCS to the data it sends through. And right there is where the issue begins; because my SCS is pretty much unique, and there are sensors it knows nothing about (such as SCD41; wind, rain for a start).

I will study some more and see where it leads me.

OK; Still not finished going through the documentation.

My initial impression is that it is not written at all well. There are a lot of words; but written by someone that already understands what is going on. I do not; and have no broad picture in my mind of how it hangs together. The documentation goes some way towards filling the knowledge gap; but there is a lot of space where there is need to interpolate.

What is missing is a task oriented list of things to do; and it would benefit from an Entity-Relationship type of diagram that shows the relationship between various blocks of (json) data.
Starting at the ‘sensors’h’ files (containing the ID values of each sensor as used in the Readings sent through to the platform.)

The documentation refers to a yellow sticker on the cover of the SCS enclosure; indicating the hardware configuration Identifier. Well my kit does not have one of those. Instead I have only the 4 digit serial number of the kit; appearing in several places. I guess it makes some sense. The work to generate a matching skeleton file has not yet been done; as my kit is unique; having sensors not found on other kits.

So, I guess my first step is to generate a candidate for a skeleton file. I can store it on my own github repository. : which is what I will do next (unless something else occurs to me)

Okaaay,

so I decided to jump in and do what seems plausible.

I have downloaded one of the JSON files sourced from:Here

working in the [Sensors] section only; I have removed entries for sensors that are not present in this particular instance of SCS.

I have added entries for new devices such as SCD41; Wind, Rain, (etc).

While I was at it; I decided to go ahead and include the EBS160/EBS210 devices also. I can do that because I have curtailed the 2nd OLED because it just does not appear to work; and this leaves me a spare Mux Channel.
So, I have now also fully integrated these two devices; giving another 6 reading types.

The ENS160 device is a VOC sensor; that has improved specification over the CCS811 device. It also gives usable data after a delay of only 3 minutes. I was gifted an evaluation kit for this device in mid-2021 and the eval kit comes with both ENS160 and ENS210. The ENS210 is a temp + RH sensor; highly accurate. The ENS210 device is read during a reading of ENS160 and the Temp + RH values are fed into the ENS160 used for calibration; making it more accurate.

After I made the integration for the two devices I was happy to find that the code compiled correctly first attempt and that readings became available immediately. Such was the experience gained during earlier work.

I will place the modified file into its own repository within my GitHub, and share it with you.

I will look and see if its appropriate for me to directly link it to my SCS V3 on the platform, but otherwise I now await for you guys @oscgonfer @victor to do your thing and stitch this together.

I actually need it to be done; to eliminate as possible cause why a lockup occurs when wifi config is first loaded. And indeed, I would like to see the various readings appear on the platform.
It matters not (at this stage) whether the GAS concentration readings are based on Alpha-sense calculations or SpecSensors (if you have not yet finished that task). If I can see Gas concentrations (even if incorrect) it will demonstrate that the data flows through from end to end; and it’s another milestone. I can move on to other things whilst you finish the Spec Sensors piece at your end.

I will share the repo with you in a little while…

Thanks and cheers,

Hi @Bryn,

Sorry, but we are really saturated these days.
Steps for this last aspect to work:

  • First we need to include the additional sensors in your unit in the platform, so that they can be ingested. We need to decide how this is done, becase they are a lot of sensors and we need to decide which ones are to be sent and not, so anything from these lines will be ignored in the platform unless the sensors are there. I suggest going with Wind Speed, direction, one or two of rain and that’s it. The reason for this is the json package size. If we see there is no issue, we can maybe later on include the other metrics, but I think it’s too specific to your case (both current and ENS160/210) values
  • Secondly, the IDs that the platform give use would need to go in Sensors.h, which I see you have properly added in your version of the firmware but they might not be the same we get. We’ll see
  • Third, the sensors part in the blueprint is only for data download purposes, all the calculations are done according to the metrics part. We need to now identify: (a) which things do we need to calculate in the platform, (b) the channels needed for those calculations, (c) if those require calibrations, and (d) if there are IDs for those calibrations we can use. (a) and (b) will go into scdata repo. (c) I assume is only for SPEC sensors, (d) we have to put in the calibrations.json file. Maybe you can use the example for alphasense sensors in here and add the file to your repo, with the SPEC sensor IDs and calibration values you have.
  • Finally, we need to add a hardware descriptor file in the hardware folder that defines your unit. The name, for instance, SCAS2100BP.json, will be what we put in the postprocessing field in the kit edit view (in this case, SCAS2100BP will be what we put there)

So, if we agree that these metrics are fine for now, we can go ahead and add them to the platform. Then we will send you the IDs and we can move on.

Regards

Thanks @oscgonfer for your response.

I’ll go through them one by one.

  1. Limitations in the number of readings that can be sent. Can I assume this limit is based on MQTT packet size; something less than the TCP packet size of 2500 bytes.(?). Is it possible to make a calculation of (roughly) how many readings will fit inside a packet? The calculation can provide a target.
    My mind immediately went in the direction of possibly interleaving readings but sending them through more often; say every 30 sec instead of 60 sec.
    Yes there are some sensor readings that I feel may be limited use or duplicated. For example the SCD41 has its own temp and RH readings; but documentation says they are thrown off by temperature effects of the CO2 sensor. Thus pretty much useless.
    Also I am looking at the readings from twin PM sensors and wondering if they can be aggregated on the device instead of sending both sets to the platform. Is it simple averaging that’s being applied ?
    Really only one set of dust figures is needed.

I agree that the INA readings do not need to be sent to the platform; since they are just a convenient way to keep track of power drain on the battery. It would be far better if it were possible to include a battery meter inside the battery; but that’s another story.

  1. Sensor ID values. Yes; sensors.h is the starting point for when I start to integrate a new device; and of course there is the risk of treading on someone else’s ID. May I suggest having an accessible master list of ID values and a method for allocation of new ones instead of guessing. I just added mine at the bottom, and used the next number having counted all the rows in the array.

I have since added sensors.h data for ENS160 and ENS210.

ENS210 values might duplicate those of inbuilt Urban board sensors; I would like to see the data before deciding which set should be eliminated; or whether averaging should be applied. Thus I consider it essential that the platform should receive this data; at least for now.

  1. It’s unclear to me but did I edit the correct file ?
    I am aware I could not deal with the first part of the file; but from what I saw; most requirements were handled except for AQI calculations which I suspect are a new idea, calling for new work, but noting that ENS160 calculates it internally but only for VOC. There are several flavours of AQI of course.
    I know nothing of channels and how they work. So I’m presently leaving it up to you since I can only handle this if I have the complete picture.
    All of the new sensors require only ‘cleaning’ to remove aberrant outlier values (zeros, nulls, non numeric and extreme high values)
    Currently; the PM data requires aggregation into one set of readings; which is already present.

  2. Yes, of course a header file SCAS2100BP.json Is needed. It contains items such as those ID values that I do not understand; it’s best if you just create this file. It also contains a link to the location of the above file containing [sensors] and [metrics] data.

Concerning calculations.json file if I were to attempt to create such a file I would need to understand what are the variables used and how they go together in the actual calculations which would look something like those given in the Spec Sensors documentation. Of course those calculations are using formulae given by Alphasense but they ought to be compatible given they deal with the same mathematical problem based on the same technology.

I do not have such knowledge; I would be flying blind; it’s inefficient to pass this task to me unless you are going to spend the time necessary to fill my knowledge gap.
I previously sent you weeks ago a file containing data and connections for the spec Sensors devices. Is it enough for you to populate the calculations.json file.?

There remain open questions about the voltages I am seeing from the ADC right now. Unconnected inputs show readings of around 0.25 volts. Is it an artifice of the ADC and associated circuit : an offset voltage that applies to all inputs ? Or is it likely due to noise and could be ignored.? The point is whether it should be taken into account in calculations or not.
The Vref values should be 50% of Vcc but are not. It’s a 1 Meg ohm voltage divider connected to the input of an opAmp. I will try to measure those resistances and see what’s happening.

If I attach an oscilloscope to the wires concerned. Its input impedance (10 megohms) will affect measurement to a small degree. I can try this and see if it helps solve these questions.

Cheers

At this time my mind turns to OTHER things that could be causing the device to halt when a new config is entered.

I am flying blind on this thing because so am not completely familiar with all aspects of the code; notably the ESP.

But, it’s possible to make some educated guesses as to what could be happening.
For example; the history of this particular SCSV3: which began in the first half of 2021. It has been turned on spasmodically since then; collecting occasional readings; and with differing but increasing sensor count.
Recently; new sensors were added of which the platform has no knowledge.
I am unaware of how this is handled; but it seems like the platform must be preconfigured to handle the sensor ID’s that are presented in the readings. How does it handle sensor ID’s it does not know about ? Does it simply ignore the data? Seems like it.

However I observe that the SCS is telling me that it’s sending several sets of readings to the platform. It’s doing the same readings time and time again. Are the readings not being acknowledged and then deleted ?
Is it normal ?

When readings have not been sent (I think) they are stored in both eeprom and flash and SD card. Need to check this. But which of these storage locations are used as source for sending data to the platform.

My hypothesis is that (maybe) there are a bunch of old unsent banked up readings; and these might be choking the system.
I know for certain that the platform storage (for this kit) has a lot of isolated readings; not much good for anybody.

So I propose to take the following actions:

  1. Fix the platform config so that it’s aware of all the sensor readings that are useful to be recorded. This process is under way; @oscgonfer is helping albeit slowly due to a busy schedule. I have to wait.
    All that I can do whilst waiting is to clean up the firmware code removing debug and making small refinement’s.

There are no more firmware additions remaining to be done; but so I have in mind to move the I2C mux code into its own class for stylistic reasons only. ( It’s working fine now).

Only When platform config is done:
2. Reinitialise the kit on the platform and get a new token and identifier and link this to the new platform config right from the start.
3. Reinitialise the SCS storage locations using whatever tools are available. Delete files on the SD Card; empty eeprom if possible. Have the SCS rewrite it’s config like a new kit.
4. Bring the SCS Online with a fresh start and see what bugs it then exhibits.

It’s not such a bad thing to have to restart the SCS after entering a new config; it’s just that it normally it is not necessary. Until recently it was not necessary.

If this is the only issue it can be forgiven; but it’s essential that I discover any remaining OTHER bugs through longer testing periods. In particular identifying lock-up conditions; because that would mean the kit cannot be left unattended for long periods. The CO2 sensor is worrisome; still reading high values. I am think of ordering a replacement; pre-calibrated version.
The system must run continuously unattended for over 24 hours at a stretch because the SCS normally performs a reset in that time frame.

While the waiting at stage 1 and testing is going on; I can get the Wifi > 4G gateway going. Then it can be placed inside its enclosure and attached to the SCS.

Concerning the ADC Board and Gas Sesnors.

I made two sets of measurements using GW Instek MSO 2204AE Oscilloscope with 1x 200 MHz passive probe. Input impedance 1MOhm // 16 pF. <<. It’s low enough to load the high impedance signal.
I also used 2 different handheld digital meters.

  1. Resistance of Vref wire to Gnd and Vcc (using Digital Meters: Inconclusive. Was supposed to be 1 MegOhms in each case. Instead I saw 0 ohms to Gnd and ~ 6 MegOhms to Vcc.
    I fear this indicates the Gas Sensor Analog Interface boards are broken.

  2. Offset voltage seen on ADC Inputs that have no sensors connected. Unequivocally there is a DC voltage of 0.2 V and a noise signal ~40 mV on top. The noise signal is in fact seen on all of the ADC wires. I live near (<1km) a military/civil airport; so there is electrical noise from Radars etc ; In addition there are several electronic devices emitting radio signals in close proximity, including the SCS itself emitting 2.4 GHz wifi signal. I am surprised the measured signal is as low as it is.

Are there any conclusions. ?

  • It is a safe bet that there is > 50% probability that the Gas ASB Interface boards are broken. I must decide whether to also order the sensors themselves; it doubles the cost. On past performance; the replacements should be here in a week or so.

  • Can we carry on with work to support Gas Concentration voltages to ppm values ? Yes. The voltages are within the expected range.

  • Does that ADC Board contribute an Offset voltage? Looks like yes. But it is applied to both Vgas and Vref the same; and so I think that the offset should fall out in the equations; which simply compare the two voltages between each other.

I have an idea concerning the gas sensors.

There are two ADC Boards; each with two ADS1115 chips;; providing a total of 16 distinct single ended Measurement channels; of which we are using only 10 to connect gas sensors.

What if we connected Vcc to one spare input, and Gnd to the other; and took a double ended measurement; which should equal the voltage we know; being 3.3 volts.
If its not; thats an error in measurement; and the difference can be applied as a correction to each measurement channel.
This could be done before readings are sent to the platform; but it would mean it is unnecessary to apply any calculated or configured offset to measurement voltages; and it removes all doubt as to the veracity of readings.

What say you on this @oscgonfer ?

Concerning Sensors for this system and “too many readings”:

We need to decide how this is done, becase they are a lot of sensors and we need to decide which ones are to be sent and not, so anything from these lines will be ignored in the platform unless the sensors are there. I suggest going with Wind Speed, direction, one or two of rain and that’s it. The reason for this is the json package size.

OK: I have gone through the [sensors] block in the Blueprint file.
Firstly I have no idea what target I am chasing. How much is too much ? How many are OK ?
I can (now I have done so) reduce the number by a few, but in fact I actually need to see the data for some of the sensors to make a decision.

For example: CS811 measures eCO2 and tVOC; but the sensor appears to be dysfunctional; slow, and others have complained about it on this forum.
ENS160 represents one of the may alternatives. I found it when enquiring with ScoSense about which Gases are measured by CCS811. They cannot tell me (they bought the product from someone else, but they offered me the ENS160/EBS210 to try out as an alternative. And thats what I intend to do. Basically put the readings from CCS811 and ENS160 alongside one another; see if they correlate. If they do, then ditch the CCS811 because its unreliable to start up. ENS160 I have already discovered was easy to implement and worked the first time I tried it. That means its more likely CCS811 will go than the ENS160. If you press me then I will eliminate CCS811, not ENS160.

Concerning Rain: Yes: the device puts out several Rain Interval Readings; of which I think that only rain interval (immediate time window) and Rain Per Interval (rain rate over the last hour) are useful. So, I have eliminated the 3 other measures for now. They will still be collected, just not sent to the platform.

Concerning INA: The reason for including it in this system is because the system will run on battery (it is now); and the amount of current drawn determines battery life, its an easy calculation. Its easier for me to display on the platform; but I can get the amount of current drain by checking using the UI.

Concerning PM readings. There are two devices generating 3 readings each that are sent to the host and there I am not certain what happens. What I think should happen is that the two sets should be averaged. What I think is happening is that the ‘min’ value is being picked:

 "EXT_PM_1_CLEAN": {
      "desc": "PM1 calculated based on both PMS5003 PM1 inputs",
      "kwargs": {
        "factor": 0.3,
        "limits": [
          0,
          1000
        ],
        "names": [
          "EXT_PM_A_1",
          "EXT_PM_B_1"
        ],
        "pick": "min",
        "window_size": 5,
        "window_type": null
      },
      "process": "merge_ts",
      "units": "ug/m3",
      "post": true,
      "id": 89
    },

If so then I think it needs to change. Not sure how to accomplish it; but it is possible to do this on the device itself fairly easily; then send only one set of readings through; eliminating a further 3 readings. If done on the device; then only ‘clean’ post processing is needed.

Concerning Gas Sensor Readings:
I have made some changes to the blueprint file and included a linkage from the [metrics] section to the [sensors section] like this:

"CO_WE": {
      "desc": "CO working electrode raw value",
      "id": "X001",
      "kwargs": {
        "channel": "ADC_4A_0"
      },
      "post": false,
      "process": "channel_names",
      "units": "V"
    },

(I added the ‘channel’ reference’)
I am uncertain what other parameters need to change; as I do not know how this is used in code (the logic).

Thereis another related piece of this puzzle:

"O3": {
      "desc": "Calculation of O3 based on AAN 803-04",
      "id": 157,
      "kwargs": {
        "ae": null,
        "alphasense_id": null,
        "from_date": null,
        "timezone": null,
        "t": "EC_SENSOR_TEMP",
        "to_date": null,
        "we": null
      },
      "post": true,
      "process": "alphasense_803_04",
      "units": "ppb"
    },

It refers to EC_SENSOR_TEMP; which is a back reference to Alphasense PT1000 device; not fitted to this system. Instead we intend to use PM_DALLAS_TEMP. Since this device reads temperature directly, I am uncertain if it can be entered directly to replace the EC_SENSOR_TEMP reference or whether there is another step.
Note: If some issue arises from using the Dallas Probe then there is in fact a temperature sensor on each Spec Sensors ASB, it would need to be wired via a separate ADC channel and indeed, voltage readings would need conversion to deg C in that case. That wiring would be messy. But lets not get ahead of ourselves, we are going with Dallas unless it proves problematic. It should be more accurate than any thermistor.

Finally, concerning calibrations file; having looked at it; as aforesaid; I would not dream of entering parameters into this file without understanding where they plug into the formulae; how they are used. Send me the formulae and I can look at it; otherwise its your job.

As it stands; we have the following data:
(a) Sensitivity Code n nA/ppm
(b) TIA Gain in Kv/A
(c) Calibration Factor in V/ppm
(c) Vgas
(d) Vref
(e) Temperature.
(f) Its also possible to compute Cross sensitivity. It is provided by Spec Sensors in their data sheets. Examples;
SO2 Sensor is also sensitive to H2S (equally sensitive). Luckily we measure both, and can compensate.
Ozone sensor is equally sensitive to Nitrogen Dioxide, but Hydrogen Sulphide reduces O3 Reading.
NO2 sensor has no cross sensitivity
H2S sensor has 1 in 25% (slight) cross sensitivity to each of SO2 and CO
CO sensor has no cross sensitivity

Effects if not x-sensitivity is not compensated.
If H2S is present; then I would expect that SO2 sensor would also light up; but O3 sensor would darken.
If NO2 is present then O3 sensor will light up too.

So, cross sensitivity is going to need a kind of matrix calculation.

{ I have not included mention of gases we do not measure therefore unable to compensate for}

I found a couple of mistakes in the ‘specification’ document I assembled for Spec Sesnors; so I will correct those and add in the Cross Sensitivity data too.

To help things along; I will also assemble a lookup table for temperature compensation for each sensor; which involves taking data off the graphs. For Taiwan; we need only to cover temperature range from -5 degC to 45 degC.

Here is a summary of recent activity concerning the ADC Board and Spec Sensors Gas Sensors.

I saw some anomalous readings output from the SCS UI interface.
There was concern that the Analog Sensor Boards had been smoked by accidental application of 5 Volts Vcc instead of specified 3.3 V supply; and the anomalous readings derived from that incident.

There are two outputs from each gas sensor that are being measured.
Vref is a reference voltage that is approx 50% of the supply voltage.
Vgas is the voltage that is converted from current output of the Gas Sensor and amplified.

The Vcc supply is around 3.3 V; therefore Vref expected to be ~1.5 V but it was over 2 volts.
I got some help from the manufacturers Engineer who supplied me the circuit diagram.

I conducted measurements using a Digital Voltmeter and Oscilloscope.
One fault was found to be that Vref is connected to Pin A0 of the ADC instead of Pin A1 expected. The source of this error is irrelevant; but it affects calculation of results and configuration of Platform json files.
The voltage on Pin A0 is within the range expected for Vref on all of the sensors; making some allowance for different bias voltages one each type of ASB (3 types).

Measurement also showed unexpected frequent variation in Vgas signals. Gas Sensors readings of course should vary with exposure to the target gas; but this is not something that occurs quickly in normal settings.
The testing showed that a spurious signal of approx 50 Hz plus noise at a level 1 - 2 Volts is periodically seen on the Vgas sensor connection to ADC. It is a burst of signal lasting a second or so, followed by silent intervals.
I searched for similar signals within the Smart Citizen system.:

  • Voltage Supply ? : In fact the SCS was supplied from a battery during the testing period. No mains supply to provide a source of ‘ripple’ sometimes seen on unregulated power supplies.
  • Serial Communications lines ? : There are 5 serial comms lines in SCS (2 x PM Sensors; Dallas Probe; Wind and Rain Sensors). None of these were operating during the testing.
  • I2C Bus ? The I2C bus showed occasional activity; but interfering signal from I2C was not seen on the Vgas lead and its timing did not coincide with the pulses of spurious signal.

So, I am forced to speculate that the spurious signal originates from remaining possibilities:
(a) A ‘parasitic’ oscillation originating from the ASB Board itself. This seems unlikely; mainly because the oscillation occurs in pulses; parasitic oscillation would likely be a continuous signal.
(b) External, nearby source of EMI. This seems most likely; because the location of my ‘lab’ is nearby an airport that is used by both military and civil air traffic. They have Radar installations that ‘sweep’ by rotating an antenna. This is a good match to the characteristics of the observed spurious signal.

The question is what to do about it. It is clearly interfering with the measurements.
If the signal originated from a wide-band radar signal; then its possible the signal level could be reduced by installation of “Ferrite Bead” type of inductive devices. The inductive impedance increases with frequency. These are installed on signal wires and effectively provide a low pass filter in conjunction with any existing decoupling capacitors and RC elements.

The idea is to install the ferrite beads on individual wires found to be susceptible to interference either source or sink.
A disadvantage of using standard ferrite beads is that they are most useful at high radio frequencies; 100 MHz and above. I do not know the centre frequency of the interfering signal (I think the 50 Hz is a sub-harmonic). Published information about airport radars indicates frequencies between 1 GHz and 6 GHz are often used; But I know from life experience that an audible signal can often be heard on AM radios much lower than that. A 1 Ghz and above Radar signal should be effectively blocked by a ferrite inductor.

Other measures that can be taken are: ‘average’ the Vgas readings over several samples. The interfering signal is superimposed on the Vgas DC signal; thus both increasing and decreasing measurements; depending on the time-instant and time-window-duration of signal sampling by the ADC. One might hope that averaging the signal readings over time will pick up both positively influenced and negatively influenced samples in even amounts. But it’s a game of chance and the effects likely random. The more samples that can be averaged; the more accurate will be the result; reflecting mostly the average gas conc. level.

I will discuss this with the smart citizen team to decide on the best approach (including whats actually feasible in the platform).
@pral2a1 @oscgonfer @victor

I also ask myself WHY is this affecting the Vgas signal and not other wires/signals such as Vref.
Well I think that it is possible the signal is being pickup by the gas sensor plates; perhaps there is a resonance with the physical dimensions of the sensor plate at the frequency used by the Radar; they act like a dipole antenna. The small signal then amplified by 500 in the op-amps forming the ASB buffer amp.
Bear in mind that Airport Radar operates at KiloWatt power levels: very strong signal nearby.

I will order some ferrite beads from Digikey; purchase from local Taiwan electronics shops is not possible right now due to the lengthy public holidays of Chinese New Year. Luckily they are fairly cheap to buy.

I have moved away from the idea of using ferrites by themselves on the Vgas leads.

Main reason is I took a look at circuit diagrams for both ADC and ASB boards. That darned Vgas wire is just a wire that runs from output of an op-amp in the ASB to the input of the ADS1115 which is high impedance. No passive components at all; no resistors, no capacitors.

If I just add the ferrite it effectively adds series inductance to the wire which will increase its “resistance” with frequency; but that resistance starts at a low value. It can work like a low pass filter but only with existing passive components providing a pathway to Gnd for the spurious AC signal and there are none. I might get some of the attenuation I need but not enough probably.

I looked for EMI filters and the like but found none suitable for filtering out the low frequency component of the spurious signal.

So, now I am looking at adding passive RC components to the connector plug : something like 330 uF and 100 ohms series resistor will attenuate signals from ~30 Hz and upward to get attenuation of something like 20 dB @ 50 Hz. It should work and I can breadboard it to confirm.

Does anyone have other suggestions ?

The spurious waveform has a period of ~25 mS per cycle = 41 Hz and the period of the burst is 1.25 Sec long with gaps in between (approx) 2 Sec.

The peaks of the burst are phase modulated with a repeating pattern (looks like data). This shows in the time domain zoomed out view, but appears like jitter when zoomed in …

The amplitude of the signal is ~160 mV (varies).

The amplitude of the signal is the same for 3 gas sensors; but the superimposed data modulation mentioned above has a different pattern.
Here is an image of the waveform to allow readers to visualise it.
The image shows waveforms emanating from two gas sensors Vgas leads.

OK… So Today was the end of Chinese New Year holiday. My parcels from Digikey arrived and I was able to visit the local Electronics shop. My new Spec Sensors Gas sensors also arrived.

  1. Fabrication of RC Low pass filter. There are a lot of steps involved that I will not describe here, but its sufficient to say that I arrived at suitable values of 820 ohms and 22 uF that achieved a ~20 dB reduction in the level of the spurious 41 Hz signal previously described. It is enough of a reduction to eliminate this source of variations in gas sensor readings. In fact the filter also attenuates much higher frequencies up to 10 MHz. In addition the effect on DC voltage measured at the input to the ADS1115 was also about 0 volts. This was based on measurements made using signal generator in the Oscilloscope; and also the ‘Frequency Response’ software function (which plots 20 Hz - 25 KHz Bode Plots).

Thus the filter met its design goals and so moving on to step 2:

  1. I made a temporary installation of the filter in line with the H2S gas sensor (picked at random); and verified that the spurious signal was still present; and that the filter attenuated the real signal.

  2. Because it was necessary to disassemble the gas sensor -+ADC + OLED board assembly which is tedious; I took the opportunity to open the ‘new’ H2S sensor and check it out, comparing with the old one and see it reflects damage or not. I installed it in place of the original that had been used for testing earlier. It looks like the old one is not damaged. At least the voltage level of Vgas is the same on the new sensor as the old sensor. The old sensor was reinstalled.

This test was done with only the ADC Board and the one H2S sensor connected; all other gas sensors and devices on the I2C I2C Mux distribution were disconnected (due to the disassembly).
When I reconnected the Oscilloscope; I was surprised to find that the spurious signal had disappeared.
you know what I was thinking at this point ! But I suppressed emotions and carried on methodically…

So, I proceeded to reconnect items one by one to see if the object emitting the spurious signal would show itself.

Only when I reconnected the NEOM8U GPS signal and waited for the Data Board firmware to initiate GPS did the 41 Hz signal reappear.

Now, this is awkward. Firstly I do not have a replacement for it. Secondly this device is used to augment the 3.3 Volt power supply supplied by the Data Board; which does not have enough capacity to run the entire system by itself. Without the help from the GPS board the devices on the I2C bus croak when the OLED is connected. There is risk of damage to the Data Board power 5V/3.3V translator circuit when it is run for a long time overloaded.
It’s also a bit of a mystery: The GPS is a Radio Receiver at very high frequencies; transmission of a low frequency 41 Hz signal is entirely unexpected.
I have to do some research as to what is the origin and purpose of this signal; see if it can/should be disabled; or if somehow it indicates a fault in the device.

If I can eliminate this signal; does this negate the need for the Low pass filter (??): I do not think so. Using the oscilloscope I still see a number of noise and mains ripple on the Vgas lead from the sensor. I think the sensor plate acts like an antenna picking up any noise that happens to be around the place. This noise can be at any level; and the high amount of gain in the Analog Sensor Board(s) means the noise is amplified, and may then interfere with actual readings which are DC.
Concerning the LPF: I now need to work out the best way to install the 0.25 watt resistor and Electrolytic capacitor at either end of the little cables joining ASB’s to ADC’s. They were neat and tidy; now they will not be so.

@oscgonfer @victor

Today I find myself at an impasse. Trying to work out which task is the logical next step.

  1. The source of pulse 41Hz spurious signal has been pinpointed to the Neo8mu GPS. I have reached out to the manufacturer support to ask them to tell me if and how it can be turned off in firmware. So far one response indicating they cannot properly read and understand English.
  • The issue here is that it’s easy to solve the noise problem by simply eliminating the gps from the design. But it’s not as easy as it seems because:
    a. GPS was included to have geographic location of readings recorded inline with sensor data. I am loath to remove it and dispense with this essential data.
    b. There is not enough 3.3 v power to run all of the devices in the system without it. In particular; without the gps there is not enough PoweR to run the OLED display.

  • So, to break that impasse I have ordered some Sparkfun Buck-Boost DC-Dc converter boards that can supply 3.3 v at 2 amps: enough to supply the whole system. The plan is to connect this supply to the system via the Grove I2C wiring. I just plug it in to spare grove sockets carefully selected to provide minimum voltage drop to all the other devices; likely connect at the I2C Mux.

  1. Other tasks that need doing need to be prioritised:
  • install the LPF(s) onto each gas sensor cable in some neat arrangement that doesn’t look too much like the hack that it is.

  • Install and test the new power board

  • experiment a little with NEO. GPS class code and try out likely library function candidates to disable that time signal causing the issue. Not wanting to spend a lot of time on this without some signal from Neo support. If you look at the library you will see why. There are literally hundreds of functions that are poorly documented and none explicitly named for the signal that appears to be causing the problem.

  • continue to patiently wait for Fablabs to do their end of the platform integration and display of all the sensor readings. When it’s ready: test and verify. As far as I am aware I have provided all the info that I am able to.

  • reassemble the SCS and continue testing and debug the firmware.

  • get the Wifi- 4G gateway working; then install into its enclosure and attach it to the main SCS enclosure.

  1. I have to say that this continuing series of impediments is taking its toll on my enthusiasm. I had hoped to have this system ready for deployment 9 months ago ! Whilst I have learned a lot I have to say it looked a lot easier going into the project than it has turned out to be. I just hope that my experience can be helpful to others and certainly the added sensors and capabilities of SCS should prove to be useful as time goes by.