Getting Data In
Highlighted

JSON Parsing error

Explorer

Setup Splunk monitoring to watch a directory. Files started coming in but with the timestamp not being parsed correctly. I adjusted by Settings > Data > Source Type then I cloned the default json and clicked Advanced and set the timestamp to this `%d-%m-%Y%H:%M:%S` for the field systemTime. (I even tried adding surrounding quotes at one point)

Example dataset:
[{
"systemTime" : "22-01-2019_15:05:01",
"fieldType" : "XXX-XXX",
"fieldLocation" : "XXX1",
"fieldCommand" : "XXXXXX",
"kernalName" : "Linux",
"nodeName" : "x86_64",
"kernalRelease" : "4.15.0-43-generic",
"kernalVersion" : "#46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018",
"machine" : "x86_64",
"processor" : "x86_64",
"hardwarePlatform" : "x86_64",
"operatingSystem" : "GNU/Linux",
"timeup" : " 15:05:01 up 8 days, 4:48, 2 users, load average: 0.35, 0.40, 0.31",
"soft1Version" : "XXXXX",
"soft2Version" : "XXXXXXXX"
}]

I noticed the files stopped coming in so I checked index=_internal source=*/splunkd.log OR source=*\\splunkd.log | search *system* log_level=ERROR and found errors like ERROR JsonLineBreaker - JSON StreamId:3524616290329204733 had parsing error:Unexpected character while looking for value: '\\'.

Despite the files not being ingested, when I go to Settings > Data Inputs > Files & Directories the file count for that directory continues to rise.
It seems to be that if I remove the timestamp part, the file does get correctly processed but _time becomes 1979...

0 Karma
Highlighted

Re: JSON Parsing error

SplunkTrust
SplunkTrust

Hi,

Please try with below configuration in props.conf for your new sourcetype.

props.conf

[yourSourcetype]
INDEXED_EXTRACTIONS=JSON
KV_MODE = none
TIMESTAMP_FIELDS=systemTime
TIME_FORMAT=%d-%m-%Y_%H:%M:%S
0 Karma
Highlighted

Re: JSON Parsing error

Explorer

Where do I place the props.conf file? I tried making one in $SPLUNK_HOME/etc/system/local and it wants me to be root to create the file. Will this cause any permissions problems? Thank you!

0 Karma
Highlighted

Re: JSON Parsing error

SplunkTrust
SplunkTrust

Does your splunk instance running as root ? If not then it should not prompt you to create file as root. You need to create file with same user as splunk is running.

You can create this props.conf in $SPLUNK_HOME/etc/system/local or if you have any custom app then $SPLUNK_HOME/etc/apps/<CUSTOM_APP>/local

0 Karma
Highlighted

Re: JSON Parsing error

Explorer

None of the events are showing. I created props.conf in /opt/splunk/etc/system/local as root and saved it as system_json1. I made sure to restart Splunk after this. However, I do not see this new source type via the GUI. Even the old events that had the time incorrectly processed disappeared.

0 Karma
Highlighted

Re: JSON Parsing error

SplunkTrust
SplunkTrust

How are you ingesting data into Splunk ? And configuration which you recently created will apply to new data only, it will not apply to data which is already ingested.

0 Karma
Highlighted

Re: JSON Parsing error

Explorer

I am ingesting the data into Splunk by using Settings > Add Data > Monitor > Files & Directories. I can change the source type for that data by Settings > Data Inputs > Files & Directories. I clicked on the directory that is causing me problems and changed the source type to props.conf file and the previous data also disappeared with that.
To be clear, Splunk is still showing that the number of files for that directory increment. For whatever reason, the files are not being processed.

0 Karma
Highlighted

Re: JSON Parsing error

SplunkTrust
SplunkTrust

So are you using same sourcetype for previous data and new data ? If you have test instance then I highly recommend to test this in Test Instance.

0 Karma
Highlighted

Re: JSON Parsing error

Explorer

Yes, I am currently on the Test Instance. The data source was new and that is why I am just now addressing the incorrect time formatting. I do not know of a way to change the sourcetype using Monitoring without it affecting both the old and new data as it only allows you to specify one.

0 Karma
Highlighted

Re: JSON Parsing error

SplunkTrust
SplunkTrust

Try to remove KV_MODE = none from props.conf and then try again.

0 Karma