topic Re: JSON Parsing error in Getting Data In

JSON Parsing error

ellothere — Tue, 22 Jan 2019 22:26:57 GMT

Setup Splunk monitoring to watch a directory. Files started coming in but with the timestamp not being parsed correctly. I adjusted by Settings > Data > Source Type then I cloned the default json and clicked Advanced and set the timestamp to this `%d-%m-%Y%H:%M:%S` for the field systemTime. (I even tried adding surrounding quotes at one point)

Example dataset:
[{ "systemTime" : "22-01-2019_15:05:01", "fieldType" : "XXX-XXX", "fieldLocation" : "XXX1", "fieldCommand" : "XXXXXX", "kernalName" : "Linux", "nodeName" : "x86_64", "kernalRelease" : "4.15.0-43-generic", "kernalVersion" : "#46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018", "machine" : "x86_64", "processor" : "x86_64", "hardwarePlatform" : "x86_64", "operatingSystem" : "GNU/Linux", "timeup" : " 15:05:01 up 8 days, 4:48, 2 users, load average: 0.35, 0.40, 0.31", "soft1Version" : "XXXXX", "soft2Version" : "XXXXXXXX" }]

I noticed the files stopped coming in so I checked index=_internal source=*/splunkd.log OR source=*\\splunkd.log | search *system* log_level=ERROR and found errors like ERROR JsonLineBreaker - JSON StreamId:3524616290329204733 had parsing error:Unexpected character while looking for value: '\\'.

Despite the files not being ingested, when I go to Settings > Data Inputs > Files & Directories the file count for that directory continues to rise.
It seems to be that if I remove the timestamp part, the file does get correctly processed but _time becomes 1979...

Re: JSON Parsing error

harsmarvania57 — Wed, 23 Jan 2019 10:04:30 GMT

Hi,

Please try with below configuration in props.conf for your new sourcetype.

props.conf

[yourSourcetype]
INDEXED_EXTRACTIONS=JSON
KV_MODE = none
TIMESTAMP_FIELDS=systemTime
TIME_FORMAT=%d-%m-%Y_%H:%M:%S

Re: JSON Parsing error

ellothere — Wed, 23 Jan 2019 14:54:42 GMT

Where do I place the props.conf file? I tried making one in $SPLUNK_HOME/etc/system/local and it wants me to be root to create the file. Will this cause any permissions problems? Thank you!

Re: JSON Parsing error

harsmarvania57 — Wed, 23 Jan 2019 14:57:59 GMT

Does your splunk instance running as root ? If not then it should not prompt you to create file as root. You need to create file with same user as splunk is running.

You can create this props.conf in $SPLUNK_HOME/etc/system/local or if you have any custom app then $SPLUNK_HOME/etc/apps/<CUSTOM_APP>/local

Re: JSON Parsing error

ellothere — Wed, 23 Jan 2019 15:37:25 GMT

None of the events are showing. I created props.conf in /opt/splunk/etc/system/local as root and saved it as system_json1. I made sure to restart Splunk after this. However, I do not see this new source type via the GUI. Even the old events that had the time incorrectly processed disappeared.

Re: JSON Parsing error

harsmarvania57 — Wed, 23 Jan 2019 15:42:44 GMT

How are you ingesting data into Splunk ? And configuration which you recently created will apply to new data only, it will not apply to data which is already ingested.

Re: JSON Parsing error

ellothere — Wed, 23 Jan 2019 15:48:27 GMT

I am ingesting the data into Splunk by using Settings > Add Data > Monitor > Files & Directories. I can change the source type for that data by Settings > Data Inputs > Files & Directories. I clicked on the directory that is causing me problems and changed the source type to props.conf file and the previous data also disappeared with that.
To be clear, Splunk is still showing that the number of files for that directory increment. For whatever reason, the files are not being processed.

Re: JSON Parsing error

harsmarvania57 — Wed, 23 Jan 2019 15:52:45 GMT

So are you using same sourcetype for previous data and new data ? If you have test instance then I highly recommend to test this in Test Instance.

Re: JSON Parsing error

ellothere — Wed, 23 Jan 2019 15:58:38 GMT

Yes, I am currently on the Test Instance. The data source was new and that is why I am just now addressing the incorrect time formatting. I do not know of a way to change the sourcetype using Monitoring without it affecting both the old and new data as it only allows you to specify one.

Re: JSON Parsing error

harsmarvania57 — Wed, 23 Jan 2019 16:03:50 GMT

Try to remove KV_MODE = none from props.conf and then try again.

Re: JSON Parsing error

ellothere — Wed, 23 Jan 2019 16:34:30 GMT

Still not happening.