All Apps and Add-ons

Problem with parsing JSON from MQTT input

pawelzak
New Member

I'd like to parse and index JSON data which come from MQTT.
Lets say that (for now) it is simple time-value JSON:
{"time": "2020-04-07 16:30:00", "value": 40}

I've installed MQTT Modular Input, cloned default "_json" Source Type and named it "simple_json". Only thing I've changed was setting "Timestamp fields" to "time".

I've added new MQTT Data Input:

  • Stanza Name: simple_json_mqtt
  • Activation Key: valid key for trial version
  • Data Output: STDOUT
  • Topic Name: simplejson/1
  • Broker Host: mqtt-broker (name of docker image with Mosquitto broker)
  • Broker Port: 1883 (no security)
  • Client ID: simplejsonmqtt
  • QOS: 1
  • Set sourcetype: From list
  • Select source type from list: simple_json (other fields left blank/default)

Now I'm sending single message (using MQTTBox):
{"time": "2020-04-07 16:30:00", "value": 40}

In splunk/data/var/log/splunk/splunkd.log I can see:

04-07-2020 15:17:07.800 +0000 ERROR JsonLineBreaker - JSON StreamId:14709566222301315061 had parsing error:Unexpected character while looking for value: 'T' - data_source="mqtt://simple_json_mqtt", data_host="splunk", data_sourcetype="simple_json"

Search for sourcetype="simple_json" returns no results

Lets try with empty lines before and after json:
`
{"time": "2020-04-07 16:31:00", "value": 41}


In log:
04-07-2020 15:19:34.655 +0000 ERROR JsonLineBreaker - JSON StreamId:14709566222301315061 had parsing error:Unexpected character while looking for value: 'T' - data_source="mqtt://simple_json_mqtt", data_host="splunk", data_sourcetype="simple_json"`

Search for sourcetype="simple_json" returns:
{"time": "2020-04-07 16:31:00", "value": 41}

Ok, now lets try to send two "events" in one MQTT message (with empty line at the end):
`{"time": "2020-04-07 16:32:00", "value": 42}
{"time": "2020-04-07 16:33:00", "value": 43}


In log:
04-07-2020 15:23:21.951 +0000 ERROR JsonLineBreaker - JSON StreamId:14709566222301315061 had parsing error:Unexpected character while looking for value: 'T' - data_source="mqtt://simple_json_mqtt", data_host="splunk", data_sourcetype="simple_json"`

Search for sourcetype="simple_json" returns:
{"time": "2020-04-07 16:33:00", "value": 43}
{"time": "2020-04-07 16:31:00", "value": 41}

So i guess there is some kind of problem with LINE_BREAKER setting in source type (by default set to: ([\r\n]+))

In the real world scenario, I won't be able to control format of JSON messages put in MQTT topic:
- order of fields
- existence of fields (lets say that "time" and "value" will be always there but also other objects/arrays/simple fields may appear)
- LINE_BREAKER

It is even possible to configure input type / source type to be able to parse "anything"?

0 Karma

to4kawa
SplunkTrust
SplunkTrust

{"time": "2020-04-07 16:33:00", "value": 43} {"time": "2020-04-07 16:31:00", "value": 41}
It may be extra character, not [\r\n]+

LINE_BREAKER = }(.)
How about this?

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.