Sending MQTT readings to another platform

rfernandez · November 3, 2021, 2:15am

Hi,
I want to send the readings from the kit directly to my own platform.
I’ve already used the APIs from SCK platform, and I’m currently reading the information my Kit has uploded to SCK platform, sending it to my platform, and working with it.

Now I want to do it the other way, that is, read directly into my platform from MQTT sent by the kit, and then send the information to the SCK platform or other platform.

This is the information of the versions I’m using:

version

			Hardware Version: 2.1
			SAM Hardware ID: 111BF6AD50515157382E3120FF0F2204
			SAM version: 0.9.8-e283e58
			SAM build date: 2021-09-13T14:17:43Z
			ESP MAC address: B6:E6:2D:65:AA:52
			ESP version: 0.9.8-e283e58
			ESP build date: 2021-09-13T14:18:25Z

I’m working on Windows.

I can think of two ways of doing this:

Redirecting the traffic from smartcitizen.me to my server using some DNS redirection
But this is something I might be able to do at home, and a regular citizen user will probably not be able to do. So this will work only for the kits I have in my network. So I won’t go this way.
Changing the MQTT server configuration in the Kit.
I would rather prefer not having to burn the whole firmware for a few changes, because I’m no expert in programming, and I’m a bit afraid of having to deal with the whole firmware upgrading thing on my own right now. Unless it is the only way, of course.

So far I’ve done the following:

Using Arduino IDE
changed from the original configuration: mqttsrv -host mqtt.smartcitizen.me -port 1883
to my local broker: mqttsrv -host 192.168.0.108 -port 1883

As my server requires some authentication, and the only authentication the Kit is sending is the Token generated at the onboarding process, I used it as the authentication for my platform.

After some configuration I got the following:

SCK > mqttsrv
		Mqtt Host: 192.168.0.108
		Mqtt Port: 1883

SCK > config
		Mode: network
		Publish interval (s): 60
		Reading interval (s): 30
		Wifi credentials: mylocalnet - password
		Token: sckToken
		Mac address:  themacaddress

When the kit reads data it shows the following:
	SCK > 
		-----------
		2021-11-03T00:28:12Z
		Temperature: 22.04 C
		Humidity: 79.99 %
		Battery: 99 %
		Light: 89 Lux
		Noise dBA: 42.52 dBA
		Barometric pressure: 100.97 kPa
		VOC Gas CCS811: null ppb
		eCO2 Gas CCS811: null ppm
		-----------
		(2021-11-03T00:28:12Z) Readings saved to flash memory.
		Recent user interaction, it's time to publish
		Connecting to Wifi...
		ESP on...
		ESP finished booting
		Synced config with ESP!!
		Connecting to Wifi...
		Connected to wifi!!
		(2021-11-03T00:28:12Z) Sent readings to platform.
		{t:2021-11-03T00:28:12Z,10:99,14:89,55:22.04,56:79.99,53:42.52,58:100.97}
		Network publish OK!! (6 readings)
		ESP off...

So, apparently, it is publishing the readings.
The problem I have right now is that I’m not receiving anything on my platform. And I don’t know how to debug or trace this to get a hint of what could be happening.

To test the platform is working fine I’ve done the following: (from the command line)

mosquitto_pub -d -q 1 -h "192.168.0.108" -t "v0/devices" -i sckToken -m {70:29}

and it works perfectly.
I’ve used a number in the key, because I’ve seen the kit has the sensors encoded and when it sends the key/value pair it just sends the sensor number and not a string key.
Also I have created a topic in the server similar to the one SMC publishes to, as I couldn’t find a way to change the topic from the shell.

So, in summary:

I think I’ve configured the Kit to point to my server, and at least the Kit believes it is sending the information there .
I’ve configured the server to receive the readings, and tested it with another MQTT client, and it works fine.

I don’t know where to go from here, so pleeease heeeelp!! I will really appreciate it !!!

Thank you very much !!!
Warm regards
Ricardo

cyrille.mdc · November 3, 2021, 1:25pm

Hi Ricardo,

Have you read this thread. I went through something similar.

It looks like you’re already using the recent SAM and ESP, so that’s good.

Try the following as a first step as it listens to everything rather than a specific:
mosquitto_sub -h 192.168.0.108 -t '#'

Otherwise see the other steps I followed in the above link and let us know what you see.

Cheers,
Cyrille

rfernandez · November 3, 2021, 5:11pm

Hi Cyrille,
Thank you very much for your answer.

Yes, I have read it!
The symptoms are quite the same.
Although there are some differences in our scenarios, that have me a bit confused.

I’m not using mosquitto as the broker, just as a client to test the broker; my broker is Thingsboard, and it is running inside Docker in the same machine. It exposes port 1883 for MQTT and 9090 for TCP.
I believe that your RPi is my Thingsboard inside Docker.

I’ve tried mosquitto_sub -h 192.168.0.108 -t “#’” and I get:
Connection error: Connection Refused: not authorised.

At this point, I’m a bit lost with the networking stuff. So I’m not sure if the refusal is a networking thing, or it is that it just doesn’t apply for my case as I’m not using mosquitto as the broker.

I tried monitoring with tcpdump, filtering by host and port like this:
tcpdump host 192.168.0.108 and port 1883 -i \Device{7224EAA9-CD46-482C-882D-2159423E5DE5} (which is the network card I use)

And I get:

13:18:22.998933 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [S], seq 6509, win 2144, options [mss 536,nop,nop,sackOK], length 0
13:18:23.006006 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [.], ack 2929932618, win 2144, length 0
13:18:23.006742 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [P.], seq 0:20, ack 1, win 2144, length 20
13:18:23.069000 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [P.], seq 20:164, ack 5, win 2140, length 144
13:18:23.138286 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [P.], seq 164:292, ack 5, win 2140, length 128
13:18:23.637709 IP 192.168.0.117.57393 > host.docker.internal.1883: Flags [P.], seq 292:420, ack 5, win 2140, length 128
13:19:28.226410 IP 192.168.0.117.62422 > host.docker.internal.1883: Flags [S], seq 6509, win 2144, options [mss 536,nop,nop,sackOK], length 0

From which I can’t get any useful information.

The thing is that I also haven’t been able to capture the traffic I send when I test the broker using the: mosquitto_pub -d -q 1 -h “192.168.0.108” -t “v0/devices” -i sckToken -m {70:29}

And I’m also sending telemetry from another simulator I have, which is successfully posting information to the broker through MQTT to port 1883. That is, I run the simulation, it generates the data, sends it to the broker (the one inside Docker), and the broker receives it fine. (similar to what you did with your Python script)
But I can’t capture it in the network either.

Well, obviously my network monitoring skills are not enough to troubleshoot this. So I’m stuck here.
Just in case, I’m running Windows, which I know sometimes complicates things a bit.

Also, I’ve tried the debugging procedure by Victor, and I got the help text he mentioned, and when telneting the IP address of the kit it returns:

*** Remote debug - over telnet - for ESP8266 (NodeMCU) - version 3.0.5
* Host name: SmartcitizenAA52 IP:192.168.0.117 Mac address:B4:E6:2D:65:AA:52
* Free Heap RAM: 32032
* ESP SDK version: 2.2.2-dev(38a443e)
******************************************************
* Commands:
	? or help -> display these help of commands
	q -> quit (close this connection)
	m -> display memory available
	v -> set debug level to verbose
	d -> set debug level to debug
	i -> set debug level to info
	w -> set debug level to warning
	e -> set debug level to errors
.....

then I sent the Hello command and:

Hello sent!
Waiting for MQTT hello response...
SCK > Hello OK!!

So, according to this, everything seems to be working fine, but I can’t get the readings.
If you could think of a way for taking the next step on this, I’ll be more than grateful!!

Thank you !!!

pral2a · November 3, 2021, 6:50pm

Hi,

Quick note here:

That should show all the messages your SCK publishes to your local broker:

$ mosquitto_sub -h 192.168.0.108 -p 1883 -t device/sck/+/+

Notice the + in the topic is a wildcard following device/sck/{device_token}/{message_type}

You can read more about MQTT topics, we call them end-points, in the platform repository:

github.com

fablabbcn/smartcitizen-api/blob/master/docs/mqtt.md#end-points

# MQTT

Readings ingestion for devices above SCK 1.5 is done using MQTT

## MQTT Broker: [EMQ 2.0](http://emqtt.io)

EMQ [Documentation](http://emqttd-docs.readthedocs.io/en/latest/index.html)

## Readings Subscriptions

The new system makes use of EMQ Shared Subscription system with Load balancing. This ensures messages distribution among currently active subscribers of a 'shared' topic.

MQTT host address is held in `.env` under the `ENV["MQTT_HOST"]` variable. Although it must be provided for `'production'`, `127.0.0.1` is set by default on `'test'` and `'development'` environments if not specified.

> NOTE: It is required defining `ENV["MQTT_HOST"]` and `ENV["MQTT_CLIENT"]` for `production` as an exception will be thrown at server startup otherwise. 

## End-points

As on [mqtt_messages_handler.rb](https://github.com/fablabbcn/smartcitizen-api/blob/master/app/lib/mqtt_messages_handler.rb)

This file has been truncated. show original

rfernandez · November 3, 2021, 9:38pm

Hi pral2a,
How are you?

From what you say the command should show the messages for every token(kit) and every message type. I’m I right?
It’s just that when I execute it
mosquitto_sub -h 192.168.0.108 -p 1883 -t device/sck/+/+
I get this … Connection error: Connection Refused: not authorized.

I can telnet to that port without any problem, so the port is accessible.

How can I trace or debug this command to try to discover why is this error happening? and be able to troubleshoot it.

NOTE: Could this have something to do with the fact that my broker is running inside a Docker container? It exposes port 1883 to the host for MQTT, and I have used it without problem with the mosquitto_pub command, and with some other tools.

Thank you !!!

rfernandez · November 3, 2021, 10:49pm

Well,

I made it work (mosquitto).
For that, I had to run it as a service, and in the config file, define a listener on port 1883 and host IP, and enable anonymous access.

After that I issue the command:
mosquitto_sub -h 192.168.0.108 -p 1883 -t device/sck/+/+
and it appears to be listening.
But, on the other hand as soon as I start the mosquito service the Kit starts failing in publishing to the broker:

ESP on...
ESP finished booting
Synced config with ESP!!
Connecting to Wifi...
Connected to wifi!!
(2021-11-03T22:39:50Z) Sent readings to platform.
{t:2021-11-03T22:39:50Z,10:99,14:70,55:21.90,56:76.92,53:38.43,58:100.45,113:76.00,112:905.00}
(2021-11-03T22:38:40Z) Sent readings to platform.
{t:2021-11-03T22:38:40Z,10:99,14:70,55:21.92,56:76.91,53:40.31,58:100.45,113:93.00,112:1012.00}
Failed sending mesg to ESP!!!
ERROR on MQTT publish
Will retry on next publish interval!!!
ESP off...

And as soon as I stop the service it starts publishing again.

So, I guess there could be a port conflict?, as both mosquitto and the broker are trying to listen in the same port.

To be continued …
Any thoughts on this would be much appreciated!

rfernandez · November 9, 2021, 12:20am

Hi,

Further investigation took me to find out that the origin of the error I was getting in the previous post was due to the timestamp format (because the platform I’m sending to expects the UNIX format and it fails parsing the JSON)

So the current problem narrows down to the following:

Is it possible to change the format in which the SCK generates the payload timestamp?

Thank you!

victor · November 22, 2021, 11:33am

Hi,

Yes, it is possible but it involves customizing the firmware.
You need to change time format in SckList.cpp line 628.

flash.readULong(wichGroup.address + GROUP_TIME) will return you a 4 byte unsigned integer in UNIX time format, so you need to do something like this (please check this code before using it):

uint32_t unixTime = flash.readULong(wichGroup.address + GROUP_TIME);
sprintf(buffer + strlen(buffer), "{t:%lu", unixTime);

If you want to be able to use the debug code you also need to change line 633, or you can just comment it.

Hope this helps!

Cheers
Victor

rfernandez · November 22, 2021, 12:30pm

Hi Victor,
Thank you very much for your answer!
I’ll study it, and give it a try.

Warm regards
Ricardo