ERROR JsonLineBreaker

[anayar]

Hi all,

I don’t know exactly how long this has been going on but I noticed today that the following error is being spammed into the /opt/splunkforwarder/var/log/splunk/splunkd.log file on our system.

04-22-2019 17:36:10.474 -0700 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '4' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="smn-sn-util01", data_sourcetype="splunkd"
04-22-2019 17:36:10.474 -0700 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '4' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="smn-sn-util01", data_sourcetype="splunkd"

There's also this error which occasionally pops up in the log but not nearly as frequently as the one above:

04-22-2019 17:47:05.009 -0700 ERROR JsonLineBreaker - JSON StreamId:8970828008188520838 had parsing error:Unexpected character while looking for value: 'A' - data_source="/var/log/messages", data_host="smn-sn-util01", data_sourcetype="syslog"
04-22-2019 17:47:05.009 -0700 ERROR JsonLineBreaker - JSON StreamId:8970828008188520838 had parsing error:Unexpected character while looking for value: 'A' - data_source="/var/log/messages", data_host="smn-sn-util01", data_sourcetype="syslog"

I tried stopping splunk, removing all the splunkd.log* files, and then restarting splunk but the error continues to show up in the logs. Any ideas as to what may be causing this?

[somesoni2]

You removed splunkd.log file from host smn-sn-util01? Also, you can run btool command on that host for splunkd sourcetype and provide output here?

/opt/splunkforwarder/bin/splunk btool props list splunkd --debug | grep -v system/default

[anayar]

Yup, I removed splunkd.log from smn-sn-util01 and the issue persisted. Here's the output of the command:

/opt/splunkforwarder/etc/apps/search/default/props.conf [splunkd]
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/apps/search/default/props.conf EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) - (?P<message>.+)
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC
/opt/splunkforwarder/etc/system/local/props.conf        AUTO_KV_JSON = false
/opt/splunkforwarder/etc/system/local/props.conf        INDEXED_EXTRACTIONS = JSON
/opt/splunkforwarder/etc/system/local/props.conf        KV_MODE = none
/opt/splunkforwarder/etc/system/local/props.conf        TIMESTAMP_FIELDS = _time
/opt/splunkforwarder/etc/system/local/props.conf        TIME_FORMAT = %s
/opt/splunkforwarder/etc/system/local/props.conf        TZ = UTC