topic Re: How to parse JSON timestamp in Getting Data In

How to parse JSON timestamp?

rchapman2x — Tue, 29 Mar 2022 16:20:12 GMT

Here's my json example file, log.json:

{"ts":"2022-01-01 01:22:34","message":"test4"} {"ts":"2022-01-01 01:22:35","message":"test5"} {"ts":"2022-01-01 01:22:36","message":"test6"}

And here's a props.conf that at least parses the json:

[ json_test ] DATETIME_CONFIG=CURRENT INDEXED_EXTRACTIONS=json NO_BINARY_CHECK=true SHOULD_LINEMERGE=false

But when I try to get "ts" to be parsed as the timestamp, it fails completely:

[ json_test ] CHARSET=UTF-8 DATETIME_CONFIG=None INDEXED_EXTRACTIONS=json NO_BINARY_CHECK=true SHOULD_LINEMERGE=false TIME_FORMAT=%Y-%m-%d %H:%M:%S TIMESTAMP_FIELDS=ts TZ = America/Los_Angeles

I've also tried adding TIME_PREFIX = "ts":", but again, it does nothing (and turns blue in the Add Data dialog, which I think means there's something wrong).

Any idea what I'm missing?

Re: How to parse JSON timestamp

gcusello — Tue, 29 Mar 2022 07:00:06 GMT

Hi @rchapman2x,

please modify your TIME_PREFIX in this way:

TIME_PREFIX = \{\"ts\":\"

{ and " are special chars and you have to escape them.

Then, where do you located props.conf?

they must be on the Universal Forwarder (if present) and on the Indexers or (when present) on Heavy Forwarders.

Ciao.

Giuseppe

Re: How to parse JSON timestamp

PickleRick — Tue, 29 Mar 2022 07:14:01 GMT

I checked with your settings and it seems to work

Mind you that you're using indexed extractions so you're not seeing json structure but extracted fields. Also the _time field value is consistent with the ts field value.

Re: How to parse JSON timestamp

rchapman2x — Tue, 29 Mar 2022 16:18:05 GMT

Hello @gcusello ,

Changing TIME_PREFIX to match your pattern did not change the results.

My props.conf is located at etc\system\local. I'm editing it through the web GUI.

I do not know what a Universal Forwarder, Indexer, or Heavy Forwarder is. I just downloaded Splunk yesterday so this is all new to me.

What's strange is that I was sure when reading through other community questions that Splunk usually outputs some kind of error message when it can't parse a log file, but I get nothing. Just a blank panel in the GUI. It obviously refreshes when I click Apply Settings, since the column tabs briefly flash on the screen, but then it goes back to being blank. I've restarted Splunk and nothing seems to work as expected.

Re: How to parse JSON timestamp

rchapman2x — Tue, 29 Mar 2022 16:19:20 GMT

Hello @PickleRick ,

I'm glad it works for you. Unfortunately all I get is a blank screen.

Re: How to parse JSON timestamp

PickleRick — Tue, 29 Mar 2022 16:39:36 GMT

There must therefore be some difference between what you showed and your effective config. Check your settings with btool

splunk btool props list json_test --debug

Re: How to parse JSON timestamp

rchapman2x — Tue, 29 Mar 2022 16:43:34 GMT

I figured it out. After restarting splunk, it informed me that there was an issue with datetime.xml. So I found a copy on Splunk's website, downloaded it, replaced my datetime.xml, and restarted the splunk daemon. That fixed the issue. Thank you!