Getting Data In

Raspberry Pi Universal Forwarder Bug Report for splunkforwarder-8.0.3-a6754d8441bf-Linux-arm.tgz:

On a Raspberry Pi 3 armv7l GNU/Linux, INDEXED_EXTRACTIONS=JSON in the props.conf file results in unrecoverable JSON StreamId processing errors:

05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json"
05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json"
05-06-2020 17:52:07.836 +0100 ERROR JsonLineBreaker - JSON StreamId:8017092045127549753 had parsing error:Unexpected character: '5' - data_source="/opt/splunkforwarder/var/log/splunk/metrics.log", data_host="rpi3", data_sourcetype="json"

with the log expanding so quickly, it fills up the /opt/splunkforwarder/var/log/splunk/splunkd.log to maximum logrotate capacity.

Steps to duplicate bug:

  1. Install splunkforwarder-8.0.3-a6754d8441bf-Linux-arm.tgz onto a Raspberry Pi 3.
  2. Edit the /opt/splunkforwarder/etc/system/local/props.conf and add the following code:

    [default]
    SHOULD_LINEMERGE = false
    KV_MODE = none
    INDEXED_EXTRACTIONS=JSON
    NO_BINARY_CHECK = true
    TRUNCATE = 0

  3. Add a local JSON file to the splunk file monitor with $SPLUNKHOME/bin/splunk add monitor /var/log/myvalidjsonfile.json -sourcetype json -host myhost -index myindex

  4. Restart splunk.

  5. Check the file tail -f $SPLUNKHOME/var/log/splunk/splunkd.log

  6. Watch it scroll away off the screen! The errors above are reported for both metrics.log and the splunkd.log itself(!)

  7. Stop splunk.

  8. Edit props.conf again and remove the line INDEXED_EXTRACTIONS=JSON.

  9. Restart splunk.

  10. Your splunkd.log is back to normal again.

0 Karma
1 Solution

Contributor

so, i think what is happening is that you are adding that INDEXED_EXTRACTIONS=JSON to default, which will apply to every log the system is forwarding , and that includes the Splunk logs themselves (everything in $SPLUNK_HOME/var/log/splunk), which are not JSON formatted.

you are better to use a specific sourcetype for your pi logs, and add the indexed extractions to that rather than in default.

./D

View solution in original post

Contributor

so, i think what is happening is that you are adding that INDEXED_EXTRACTIONS=JSON to default, which will apply to every log the system is forwarding , and that includes the Splunk logs themselves (everything in $SPLUNK_HOME/var/log/splunk), which are not JSON formatted.

you are better to use a specific sourcetype for your pi logs, and add the indexed extractions to that rather than in default.

./D

View solution in original post

You're right, but ewwww, that's expected behaviour?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!