MQTT and changing the data server

cyrille.mdc · August 21, 2021, 5:54pm

So I’m trying to send my SCK2.1 data to another server.

Currently I’m using a RPI, where I have successfully tested the MQTT service using this tutorial.

I’m able to publish and subscribe without any problems to my tests from other machines on the network.

I’m unable to retrieve any of the published messages from the SCK2.1.

Here’s what I’ve done:

In the SCK2.1 SHELL: mqttsrv -host 192.168.2.101 -port 1883 (where 192.168.2.101 is my RPI)
I can check that the configuration functioned with the SHELL command mqttsrv and get the response:

Mqtt Host: 192.168.2.101
Mqtt Port: 1883

In the SHELL the SCK2.1 is stating it’s publishing successfully:

(2021-08-21T17:47:14Z) Sent readings to platform.
Network publish OK!! (15 readings)
(2021-08-21T17:47:24Z) Sent readings to platform.
Network publish OK!! (15 readings)

I can listen/subscribe to all publications to the mosquitto broker using the client:
mosquitto_sub -h 192.168.2.101 -t '#'

I get no messages from the SCK - except those I send myself using a python script I created to test the MQTT broker.

I’m at a bit of a loss and would love some input from someone who’s gotten the changing of the hostname working.

Cheers,
Cyrille

cyrille.mdc · August 22, 2021, 8:22am

More information - testing.

After further testing I should add that the SHELL command hello does not return any errors when pointing to my host (mqttsrv -host 192.168.2.101 -port 1883) - but it doesn’t return anything either.

I’m building from the latest github repository version.

If I reset the SCK2.1 to the default host mqttsrv -host mqtt.smartcitizen.me -port 1883 and test hello I do get a Hello sent! followed eventually, but not always, by a Hello OK!!.

It’s really the Network publish OK!! that’s confusing me. I’ll dig into the code and see what’s required for those to be sent.

cyrille.mdc · August 22, 2021, 9:34am

Continued…

I’ve turned on debug -esp.

When I enter the commands hello or publish test message, I frequently get returned:
Can't send message, ESP is off or still booting...

But not always. I’ve received successful messages from both. So I guess ESP goes to sleep (to save battery?) but can’t send my messages during these periods. This is independent of which host I set.

I think I was wrong in my above post about hello not working - it’s responding positively on either host (my test and the default) sometimes. Irregardless, my MQTT broker isn’t receiving anything.

cyrille.mdc · August 22, 2021, 10:10am

As I dig into the code it’s not clear to me that the data is being sent using MQTT at all.

bedinek · August 22, 2021, 5:41pm

Did you try to run tcpdump on RPi to check if SCK does indeed send it to RPi address?

cyrille.mdc · August 22, 2021, 8:08pm

Hi @bedinek ,

Thanks for the suggestion!

Please note that the RPi is running Pi-hole and also managing DHCP - this may have an impact on the outputs below.

My SCK2.1 responds to the mqttsrv command with:

Mqtt Host: 192.168.2.101
Mqtt Port: 1883

So I can monitor incoming connections on port 1883 on the RPi (192.168.2.101) using:
sudo tcpdump dst port 1883

Nothing appears in the tcpdump from the SCK2.1 however.

Testing a mqtt publishing python script (as described above earlier) from my Ubuntu machine (192.168.2.226) does appear in the tcpdump output however:

21:47:51.315252 IP 192.168.2.226.47815 > 192.168.2.101.1883: Flags [S], seq 1355256520, win 64240, options [mss 1460,sackOK,TS val 3881306389 ecr 0,nop,wscale 7], length 0
21:47:51.315344 IP 192.168.2.101.1883 > 192.168.2.226.47815: Flags [S.], seq 4225788666, ack 1355256521, win 65160, options [mss 1460,sackOK,TS val 822255602 ecr 3881306389,nop,wscale 7], length 0
21:47:51.316405 IP 192.168.2.226.47815 > 192.168.2.101.1883: Flags [.], ack 1, win 502, options [nop,nop,TS val 3881306390 ecr 822255602], length 0
21:47:51.316497 IP 192.168.2.226.47815 > 192.168.2.101.1883: Flags [P.], seq 1:15, ack 1, win 502, options [nop,nop,TS val 3881306390 ecr 822255602], length 14
21:47:51.316536 IP 192.168.2.101.1883 > 192.168.2.226.47815: Flags [.], ack 15, win 509, options [nop,nop,TS val 822255603 ecr 3881306390], length 0
21:47:51.316582 IP 192.168.2.226.47815 > 192.168.2.101.1883: Flags [FP.], seq 15:58, ack 1, win 502, options [nop,nop,TS val 3881306390 ecr 822255602], length 43
21:47:51.317100 IP 192.168.2.101.1883 > 192.168.2.226.47815: Flags [P.], seq 1:5, ack 59, win 509, options [nop,nop,TS val 822255604 ecr 3881306390], length 4

In the SHELL, while testing, I’ve turned on debug -esp and get messages such as:

Sending msg to ESP with 3 parts and 165 bytes
{t:2021-08-22T19:21:30Z,10:99,14:12,55:22.43,56:69.52,53:27.28,58:100.37,113:21.00,112:541.00,128:0,125:49.759043,126:6.637683,127:34.85,129:0.00,131:9999.00,130:0}
Sent part num 0
{t:2021-08-22T19:21:30Z,10:99,14:12,55:22.43,56:69.52,53:2
Sent part num 1
7.28,58:100.37,113:21.00,112:541.00,128:0,125:49.759043,126
Sent part num 2
:6.637683,127:34.85,129:0.00,131:9999.00,130:0}:EA:A0","es
Receiving msg from ESP in 1 parts
Network publish OK!! (15 readings)

I’ve also tried while debug is off as I’d read somewhere in the code it can cause problems with communication.

By looking at the Pi-hole DHCP table I can see that 192.168.2.219 is the IP of the SCK2.1.
Is there an easier way to retrieve the IP of the SCK2.1 in the SHELL?

I’ve also tried monitoring using the following command:
sudo tcpdump host 192.168.2.219 -vv

And received this:

21:50:24.737780 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.2.101 is-at dc:a6:32:c4:71:f6 (oui Unknown), length 28
21:50:24.742115 IP (tos 0x0, ttl 255, id 3, offset 0, flags [none], proto UDP (17), length 66)
    192.168.2.219.43050 > 192.168.2.101.domain: [udp sum ok] 25898+ A? mqtt.smartcitizen.me. (38)
21:50:24.742635 IP (tos 0x0, ttl 64, id 61650, offset 0, flags [DF], proto UDP (17), length 82)
    192.168.2.101.domain > 192.168.2.219.43050: [udp sum ok] 25898 q: A? mqtt.smartcitizen.me. 1/0/0 mqtt.smartcitizen.me. A 85.159.210.156 (54)
21:50:24.742942 IP (tos 0x0, ttl 255, id 4, offset 0, flags [none], proto UDP (17), length 66)
    192.168.2.219.43050 > 192.168.2.101.domain: [udp sum ok] 25898+ A? mqtt.smartcitizen.me. (38)
21:50:24.743302 IP (tos 0x0, ttl 64, id 61651, offset 0, flags [DF], proto UDP (17), length 82)
    192.168.2.101.domain > 192.168.2.219.43050: [udp sum ok] 25898 q: A? mqtt.smartcitizen.me. 1/0/0 mqtt.smartcitizen.me. A 85.159.210.156 (54)
21:50:24.746092 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.1 tell 192.168.2.219, length 50
21:50:24.746524 IP (tos 0x0, ttl 255, id 6, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.2.219 > 192.168.2.101: ICMP 192.168.2.219 udp port 43050 unreachable, length 36
	IP (tos 0x0, ttl 64, id 61651, offset 0, flags [DF], proto UDP (17), length 82)
    192.168.2.101.domain > 192.168.2.219.43050: [|domain]
21:50:28.807849 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.219 tell 192.168.2.101, length 28

I’m not very clear on what’s happening here but the fact that mqtt.smartcitizen.me is appearing suggests to me that either it’s checking with the pi-hole whether this host is on the black list and/or it’s trying to communicate with that host.

This suggests that the mqtt hostname hasn’t been overwritten, no?

I have a second SCK2.1 running stock firmware. It produces a similar tcpdump output (from another IP).

What are your thoughts on this?

pral2a · August 23, 2021, 5:44pm

Hi @cyrille.mdc

Good work so far and great debug info!

MQTT host configuration over the shell was included not so long ago in the latest release by @victor I’ll wait for him to have a look at your logs

In the meantime you can try to directly hardcode the information and recompile the firmware

github.com

fablabbcn/smartcitizen-kit-21/blob/62c2239069d8729c0fec1f3e83bbcd65e119d516/lib/Shared/Config.h#L44-L45


struct Mqtt { char server[64]="mqtt.smartcitizen.me"; uint16_t port=1883; };
struct Ntp { char server[64]="ntp.smartcitizen.me"; uint16_t port=80; };

cyrille.mdc · August 24, 2021, 5:09pm

Hi @pral2a ,

I updated my Config.h to the following:

//struct Mqtt { char server[64]="mqtt.smartcitizen.me"; uint16_t port=1883; };
struct Mqtt { char server[64]="192.168.2.101"; uint16_t port=1883; };
struct Ntp { char server[64]="ntp.smartcitizen.me"; uint16_t port=80; };

I then built and uploaded the firmware.

I used the SHELL to check that:

the firmware is running the version I just uploaded SAM build date: 2021-08-24T18:56:29Z
the mqttsrv command prints expected output:

Mqtt Host: 192.168.2.101
Mqtt Port: 1883

I then ssh’d into my RPi that has the mqtt broker (mosquitto) and used tcpdump to listen on the SCk2.1 IP address. I also reset the wifi and token with config ... and this was the output of note:

19:07:52.206791 IP (tos 0x0, ttl 255, id 9, offset 0, flags [none], proto UDP (17), length 66)
    192.168.2.219.61474 > 192.168.2.101.domain: [udp sum ok] 19493+ A? mqtt.smartcitizen.me. (38)
19:07:52.224328 IP (tos 0x0, ttl 64, id 49697, offset 0, flags [DF], proto UDP (17), length 82)
    192.168.2.101.domain > 192.168.2.219.61474: [udp sum ok] 19493 q: A? mqtt.smartcitizen.me. 1/0/0 mqtt.smartcitizen.me. A 85.159.210.156 (54)

I also listened on port 1883 but received no output.

Cheers.
C-

pral2a · August 24, 2021, 5:23pm

Hi @cyrille.mdc

Good work! Just to confirm. Did you check if you can publish on the local broker on the Pi (192.168.2.101:1883) using another client on the same LAN? You can use mosquitto_pub and mosquitto_sub or any other client you like

cyrille.mdc · August 24, 2021, 7:33pm

Hi @pral2a ,

If I publish to the RPi (192.168.2.101) from my Ubuntu desktop on the same network, the RPi receives it.

I can see this by using the command sudo tcpdump port 1883:

21:20:23.899379 IP 192.168.2.226.48235 > 192.168.2.101.1883: Flags [S], seq 1163267837, win 64240, options [mss 1460,sackOK,TS val 3918738813 ecr 0,nop,wscale 7], length 0
21:20:23.899463 IP 192.168.2.101.1883 > 192.168.2.226.48235: Flags [S.], seq 509911008, ack 1163267838, win 65160, options [mss 1460,sackOK,TS val 993408186 ecr 3918738813,nop,wscale 7], length 0
21:20:23.900543 IP 192.168.2.226.48235 > 192.168.2.101.1883: Flags [.], ack 1, win 502, options [nop,nop,TS val 3918738815 ecr 993408186], length 0
21:20:23.900632 IP 192.168.2.226.48235 > 192.168.2.101.1883: Flags [P.], seq 1:15, ack 1, win 502, options [nop,nop,TS val 3918738815 ecr 993408186], length 14
21:20:23.900670 IP 192.168.2.101.1883 > 192.168.2.226.48235: Flags [.], ack 15, win 509, options [nop,nop,TS val 993408187 ecr 3918738815], length 0
21:20:23.900749 IP 192.168.2.226.48235 > 192.168.2.101.1883: Flags [FP.], seq 15:58, ack 1, win 502, options [nop,nop,TS val 3918738815 ecr 993408186], length 43
21:20:23.901396 IP 192.168.2.101.1883 > 192.168.2.226.48235: Flags [P.], seq 1:5, ack 59, win 509, options [nop,nop,TS val 993408188 ecr 3918738815], length 4

Similarly the RPi (192.168.2.101) broker mosquitto_sub -h 192.168.2.101 -t "#" -v outputs the following topic/message pairs from various python3 mqtt publishing script calls:

topic_name message_sent
another/topic another message

I’m baffled by the source of the mqtt.smartcitizen.me url. I’ve rgreped through the whole repo and no other reference to it exists.
We agree there’s no way the NTP server is used in any way to retrieve the mqtt url?

I think I need to follow the code and see where/when the mqtt calls are being made…

victor · August 25, 2021, 11:47am

Hi @cyrille.mdc,

I wasn’t able to replicate your problem, I’m not sure what is happening on your setup. I can give you some info on the source code and some tricks on debugging the SCK to see if we can find a solution.

First about the configuration, it is stored on the flash memory of the SAM chip and sent to the ESP every time it boots, the definitions are in the config.h file.

The command that changes the mqtt server is called mqttsrv and it’s code is here, you can issue it with the -host flag to change the server or with the -port to change the port, if you issue only the command you will see the current configuration.

mqttsrv:     Configure mqtt server address and port: [-host serverName] [-port portNum]

For debbuging this type of problems the best way is to use the telnet feature:

Configure your SKC in network mode with wifi credentials and any token:

SCK > config -mode network -wifi "ssid" "password" -token aabbcc

Configure your custom MQTT server:

SCK > mqttsrv -host 192.168.2.101
Mqtt Host: 192.168.2.101
Mqtt Port: 1883

Enable telnet debugging:

SCK > debug -telnet
Telnet debug: true

Reset the SCK to be sure everything is updated

SCK > reset

Once connected to your SCK again enter shell mode with:

SCK > shell -on
Shell mode: on

The led should turn yellow. In this mode the kit will not follow the normal loop and will only react to your commands (keep in mind that some commands take you out of shell mode)
6. To be sure that your’e connected to wifi reboot the esp:

SCK > esp -reboot

Once you see the connected message check your ip with:

SCK > netinfo
SCK >
Hostname: Smartcitizen3F4E
IP address: 192.168.2.219
MAC address: F6:CF:A2:6A:3F:4E
ESP version: 0.9.2-62c2239
ESP build date: 2021-08-25T12:51:32Z

From your computer (I asume you are using linux) connect via telnet

telnet 192.168.2.219

If everything went Ok you should see some output and some help info (not directly related to the SCK but to the debug lib we use). Now you can try (via the SCK USB terminal) to execute the hello command
and check the output on the telnet terminal:

In this case the ip address doesn’t exist so the MQTTconnect function fails and report the error code that you can check in the library documentation.

Hope this helps!!
Please keep us posted of your findings.

Cheers
Victor

cyrille.mdc · August 25, 2021, 1:39pm

Thanks @victor ,

I’m fine until the step to telnet into the SCK2.1:

Trying 192.168.2.219...
telnet: Unable to connect to remote host: Connection refused

In the previous step my version of ESP is a bit older:

Hostname: SmartcitizenEAA0
IP address: 192.168.2.219
MAC address: 3E:61:05:DF:EA:A0
ESP version: 0.9.2-a91f850
ESP build date: 2019-08-20T13:17:16Z

While I updated SAM I never did for ESP - perhaps I should have.
I’m having trouble getting ESP building - so I’ll focus on that and get back to you.

Thanks!

victor · August 25, 2021, 2:05pm

Hi @cyrille.mdc,

I think all your problems will be solved with ESP firmware update!
The custom mqtt broker support is from Nov 2020.
I’ve just build ESP firmware from a clean git clone (without old libraries) and worked without problems, I hope it works for you.

Cheers!
Victor

cyrille.mdc · August 27, 2021, 11:20am

Thanks @victor !

After updating the ESP firmware did it. I’m receiving data on my server now.
Thanks!