topic Re: JSON data with date not indexed in Getting Data In

JSON data with date not indexed

osec2a — Thu, 31 Aug 2017 08:06:15 GMT

Hi,

I am trying to index JSON data but Splunk refused to index it and I have no errors in logs.
The format of my data is :

{"test1":"2017-08-31", "test2":"12.34.56"}

I tried many format :

{"test1":"2017-08-31", "test2":"12.34.56"}" => not indexed
{"test1":"12.34.56", "test2":"2017-08-31"}" => not indexed
{"test1":"2017-08-31", "test2":2017/08/3"}"  = > indexed successfully
{"test1":"2017.08.31", "test2":17.08.3"}" => indexed successfully
{"test1":"2017_08_31", "test2":17.08.3"}"  => indexed successfully

Why Splunk doesn't want to index my JSON with these data :

{"test1":"2017-08-31","test2":"12.34.56"}

Thanks for your help

Re: JSON data with date not indexed

niketn — Thu, 31 Aug 2017 10:12:57 GMT

@osec2a, I tried adding {"test1":"2017-08-31", "test2":"12.34.56"} and it worked fine for me.

I had done Single File Upload in preview mode and I just created my Custom Sourcetype under "Search & Reporting App" with index="main".

The data looked good in preview mode as well as after adding to index in Splunk Search.

Re: JSON data with date not indexed

osec2a — Thu, 31 Aug 2017 11:57:08 GMT

My data are outputed by python script.
My script output is json data.
When the script output is {"test1":"2017-08-31", "test2":"12.34.56"}, data are not indexed, but when I delete the dot ".", data are indexed.

Re: JSON data with date not indexed

cpetterborg — Thu, 31 Aug 2017 14:20:38 GMT

If you explicitly set the data format to be %Y-%m-%d, does it work in all cases? (props.conf) I suspect that by default Splunk is probably looking at 12.34.56 as a date string, and if it falls outside the range of the date you are looking at, or it is prior to 1970 (I'm assuming that 12.34.56 is a made up piece of data, but I have nothing else to go on), then you are not going to find it in your search or it is too old to be properly indexed.

Also, though it is probably just a typo, your JSON data in your example data is bogus JSON. It probably should look like this:

{"test1":"2017-08-31", "test2":"12.34.56"} => not indexed
{"test1":"12.34.56", "test2":"2017-08-31"} => not indexed
{"test1":"2017-08-31", "test2":"2017/08/3"}  = > indexed successfully
{"test1":"2017.08.31", "test2":"17.08.3"} => indexed successfully
{"test1":"2017_08_31", "test2":"17.08.3"}  => indexed successfully