Hello good people of the Splunk Community. This one's got me foxed.
I noticed this morning that the splunkd logs on my Raspberry Pi-hosted Universal Forwarder are rotating really quickly (check ou...
See more...
Hello good people of the Splunk Community. This one's got me foxed.
I noticed this morning that the splunkd logs on my Raspberry Pi-hosted Universal Forwarder are rotating really quickly (check out the timestamps below - it is literally creating a log entry as fast as the CPU will spin) and I've got no idea why.
Oddly, the error appears to originate in Splunk's own log at:
/opt/splunkforwarder/var/log/splunk/splunkd.log
At first I thought the error must have been introduced from a parsed log, but then I realised two odd things - firstly, the splunkd errors I'm seeing reference the log itself as the source of the problem and secondly it appears to take issue with the number '5' (in it's own log)?
Removing the logs and restarting the forwarder doesn't help, rebooting the RPi doesn't help. As soon as the splunkd service starts, it immediately spams the splunkd.log with this. Anyone any ideas what I'm missing?
Here's what it looks like:
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.117 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.118 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.118 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
05-04-2020 15:48:24.118 +0100 ERROR JsonLineBreaker - JSON StreamId:14919777892573414995 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/splunkd.log", data_host="rpi3", data_sourcetype="json"
... and lots more ...
... and more ...
... more ..