Getting Data In

Can you help me extract a timestamp from JSON?

Path Finder

Hi,

I'm having trouble extracting timestamps from JSON on a production environment: Timestamp field is not used by Splunk as timestamp (and Splunk used the _indextime as timestamp). I tried to configure props.conf (see below) on both indexers and heavy forwarder (data are pulled from a data store on Azure). I have to say that on a local instance, it works fine.
alt text

Here is my raw data :

{"odata.etag": "W/\"datetime'2019-01-09T08%3A29%3A09.2933828Z'\"", "PartitionKey": "201901090829", "levelno": "10", "hostname": "5c6744d1d32c", "levelname": "DEBUG", "version": "v1.2.1-741-g19422ce", "component": "Watcher", "Timestamp": "2019-01-09T08:29:09.2933828Z", "RowKey": "20190109082909284-5c6744d1d32c-22-00", "message": "Watcher running...", "pathname": "/app/ccc/watcher/perimeter.py", "lineno": "61"}

Different props.conf that I tried without any success :

[sourcetype]
CHARSET = AUTO
KV_MODE = JSON
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

[sourcetype]
CHARSET = AUTO
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

Any help is very welcome 🙂

Francois

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

View solution in original post

0 Karma

Path Finder

alt text
Thank again for your quick answer. Here is an example of what I'm facing. And I've no props.conf for this sourcetype. Maybe I should configure one indicating that it's cooked data ?

0 Karma

SplunkTrust
SplunkTrust

On search head can you please configure below props.conf

[yoursourcetype]
KV_MODE = none
0 Karma

Path Finder

That's perfect ! Thank you !

0 Karma

Path Finder

Thank you ! It works very well ! But...
Now I have multivalue fields 😕
I guess both extracted on HF and indexer. Any ideas ?

0 Karma

SplunkTrust
SplunkTrust

Only first full instance of splunk is parsing the data so in your case only HF is parsing data. Can you please post your multivalue field sample data and what problem are you facing with it ? Also I'll suggest to remove config from Indexer if HF is parsing data.

EDIT: It maybe due to config on Indexer, so first remove config from Indexer and then check.

0 Karma