Invalid JSON format in Kit's output payload

rfernandez · November 9, 2021, 12:28am

Hi,

The output payload of the device is not a valid JSON format.
I’ve tested it in several online JSON/JS parsers and it doesn’t pass the test in any of them.

This is what the Kit is sending as payload:
{t:2021-11-08T16:17:15Z,10:99,14:24,55:20.85…}

And this is what it should look in a valid JSON format:

{“t”:“2021-11-08T16:17:15Z”,“10”:99,“14”:24,“55”:20.85,…}

Is there a way to fix this?

Thank you!

pral2a · November 10, 2021, 4:33pm

Hi @rfernandez

Please, check what I posted here:

There are different topics the kit published. Kits running the latest firmware publishes batch data in a raw format under the topic device/sck/%s/readings/raw instead of the original device/sck/%s/readings. Raw means they push data directly from the internal flash memory without assembling a proper JSON file to save memory resources at the Kit side.

That’s the Ruby function that turns the custom data format into a valid JSON file. You can easily implement it in any other language.

github.com

fablabbcn/smartcitizen-api/blob/be286f9a36d153e8d62b3b40db8a000e5086342d/app/lib/mqtt_messages_handler.rb#L42-L55

    
      
          def self.parse_raw_readings(message)
            clean_tm = message[1..-2].split(",")[0].gsub("t:", "").strip
            raw_readings = message[1..-2].split(",")[1..]
          
          
  reading = { 'data' => ['recorded_at' => clean_tm, 'sensors' => []] }
          
          
  raw_readings.each do |raw_read|
              raw_id = raw_read.split(":")[0].strip
              raw_value = raw_read.split(":")[1]&.strip
              reading['data'].first['sensors'] << { 'id' => raw_id, 'value' => raw_value }
            end
          
          
  JSON[reading]
          end

You can also change the firmware to use any publishing format that you like. However, currently, we only support this one.

github.com

fablabbcn/smartcitizen-kit-21/blob/62c2239069d8729c0fec1f3e83bbcd65e119d516/esp/src/SckESP.cpp#L345-L429

    
      
          bool SckESP::mqttPublish()
          {
          	debugOUT(F("Trying MQTT publish..."));
          
          
	if (mqttConnect()) {
          
          
		// Prepare the topic title
          		char pubTopic[27];
          		sprintf(pubTopic, "device/sck/%s/readings", config.token.token);
          
          
		debugOUT(String(pubTopic));
          		debugOUT(String(netBuff));
          
          
		char pubPayload[MQTT_BUFF_SIZE];
          		
          
          
		// /* Example
          		// {	"data":[
          		// 		{"recorded_at":"2017-03-24T13:35:14Z",
          		// 			"sensors":[

This file has been truncated. show original

rfernandez · November 10, 2021, 9:04pm

Hi,
Thank you very much for your answer.
I apologize, as I feel there are some underlying concepts I’m missing.

Please let me rephrase what you said to check if I understood well: and please feel free to correct me if I misunderstood anything.

The device changed the way it publishes information. Previous versions of the firmware published to the “…/readings” topic in JSON format. Current (latest) version of the firmware publishes to the “…/readings/raw” topic that is not formated to JSON before publishing, to save resources on the device. And, it is not publishing to the “…/readings” topic anymore.

So the readings I’ll be getting are in the form that is shown in the shell:
{t:2021-11-10T19:51:33Z,10:99,14:102,55:23.40,56:58.55,53:38.62,58:101.59,113:20.00,112:534.00}

and I will get them under the “…/readings/raw” topic in my broker.

Thank you !!

P.S. I was not able to implement mosquitto_sub -h 192.168.0.108 -p 1883 -t device/sck/+/+. For some reason I was not able to determine yet, my broker denies the subscriptions from mosquito_sub; maybe has something to do with a particular Windows configuration.