topic Re: Can you help me extract a timestamp from JSON? in Getting Data In

Can you help me extract a timestamp from JSON?

francoisternois — Wed, 09 Jan 2019 08:49:26 GMT

Hi,

I'm having trouble extracting timestamps from JSON on a production environment: Timestamp field is not used by Splunk as timestamp (and Splunk used the _indextime as timestamp). I tried to configure props.conf (see below) on both indexers and heavy forwarder (data are pulled from a data store on Azure). I have to say that on a local instance, it works fine.

Here is my raw data :

{"odata.etag": "W/\"datetime'2019-01-09T08%3A29%3A09.2933828Z'\"", "PartitionKey": "201901090829", "levelno": "10", "hostname": "5c6744d1d32c", "levelname": "DEBUG", "version": "v1.2.1-741-g19422ce", "component": "Watcher", "Timestamp": "2019-01-09T08:29:09.2933828Z", "RowKey": "20190109082909284-5c6744d1d32c-22-00", "message": "Watcher running...", "pathname": "/app/ccc/watcher/perimeter.py", "lineno": "61"}

Different props.conf that I tried without any success :

[sourcetype]
CHARSET = AUTO
KV_MODE = JSON
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

[sourcetype]
CHARSET = AUTO
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

Any help is very welcome 🙂

Francois

Re: Can you help me extract a timestamp from JSON?

harsmarvania57 — Wed, 09 Jan 2019 09:38:39 GMT

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

Re: Can you help me extract a timestamp from JSON?

francoisternois — Wed, 09 Jan 2019 10:20:57 GMT

Thank you ! It works very well ! But...
Now I have multivalue fields 😕
I guess both extracted on HF and indexer. Any ideas ?

Re: Can you help me extract a timestamp from JSON?

harsmarvania57 — Wed, 09 Jan 2019 10:23:51 GMT

Only first full instance of splunk is parsing the data so in your case only HF is parsing data. Can you please post your multivalue field sample data and what problem are you facing with it ? Also I'll suggest to remove config from Indexer if HF is parsing data.

EDIT: It maybe due to config on Indexer, so first remove config from Indexer and then check.

Re: Can you help me extract a timestamp from JSON?

francoisternois — Wed, 09 Jan 2019 10:40:18 GMT

Thank again for your quick answer. Here is an example of what I'm facing. And I've no props.conf for this sourcetype. Maybe I should configure one indicating that it's cooked data ?

Re: Can you help me extract a timestamp from JSON?

harsmarvania57 — Wed, 09 Jan 2019 10:44:24 GMT

On search head can you please configure below props.conf

[yoursourcetype]
KV_MODE = none

Re: Can you help me extract a timestamp from JSON?

francoisternois — Wed, 09 Jan 2019 11:55:34 GMT

That's perfect ! Thank you !