Can you help me extract a timestamp from JSON?

[francoisternois]

Hi,

I'm having trouble extracting timestamps from JSON on a production environment: Timestamp field is not used by Splunk as timestamp (and Splunk used the _indextime as timestamp). I tried to configure props.conf (see below) on both indexers and heavy forwarder (data are pulled from a data store on Azure). I have to say that on a local instance, it works fine.

Here is my raw data :

{"odata.etag": "W/\"datetime'2019-01-09T08%3A29%3A09.2933828Z'\"", "PartitionKey": "201901090829", "levelno": "10", "hostname": "5c6744d1d32c", "levelname": "DEBUG", "version": "v1.2.1-741-g19422ce", "component": "Watcher", "Timestamp": "2019-01-09T08:29:09.2933828Z", "RowKey": "20190109082909284-5c6744d1d32c-22-00", "message": "Watcher running...", "pathname": "/app/ccc/watcher/perimeter.py", "lineno": "61"}

Different props.conf that I tried without any success :

[sourcetype]
CHARSET = AUTO
KV_MODE = JSON
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

[sourcetype]
CHARSET = AUTO
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

Any help is very welcome 🙂

Francois

[harsmarvania57 · Accepted solution]

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

[francoisternois]

Thank again for your quick answer. Here is an example of what I'm facing. And I've no props.conf for this sourcetype. Maybe I should configure one indicating that it's cooked data ?

[harsmarvania57]

On search head can you please configure below props.conf

[yoursourcetype]
KV_MODE = none

[francoisternois]

That's perfect ! Thank you !

[francoisternois]

Thank you ! It works very well ! But...
Now I have multivalue fields 😕
I guess both extracted on HF and indexer. Any ideas ?

[harsmarvania57]

Only first full instance of splunk is parsing the data so in your case only HF is parsing data. Can you please post your multivalue field sample data and what problem are you facing with it ? Also I'll suggest to remove config from Indexer if HF is parsing data.

EDIT: It maybe due to config on Indexer, so first remove config from Indexer and then check.