Getting Data In

Can you help me extract a timestamp from JSON?

francoisternois
Path Finder

Hi,

I'm having trouble extracting timestamps from JSON on a production environment: Timestamp field is not used by Splunk as timestamp (and Splunk used the _indextime as timestamp). I tried to configure props.conf (see below) on both indexers and heavy forwarder (data are pulled from a data store on Azure). I have to say that on a local instance, it works fine.
alt text

Here is my raw data :

{"odata.etag": "W/\"datetime'2019-01-09T08%3A29%3A09.2933828Z'\"", "PartitionKey": "201901090829", "levelno": "10", "hostname": "5c6744d1d32c", "levelname": "DEBUG", "version": "v1.2.1-741-g19422ce", "component": "Watcher", "Timestamp": "2019-01-09T08:29:09.2933828Z", "RowKey": "20190109082909284-5c6744d1d32c-22-00", "message": "Watcher running...", "pathname": "/app/ccc/watcher/perimeter.py", "lineno": "61"}

Different props.conf that I tried without any success :

[sourcetype]
CHARSET = AUTO
KV_MODE = JSON
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

[sourcetype]
CHARSET = AUTO
INDEXED_EXTRACTIONS = json
KV_MODE = none
MAX_TIMESTAMP_LOOKAHEAD = 20
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = Timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = Timestamp": "
category = Structured
disabled = false
pulldown_type = true

Any help is very welcome 🙂

Francois

0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

View solution in original post

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

If you are receiving data from Azure on Heavy Forwarder then try below props.conf on Heavy Forwarder.

[yoursourcetype]
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=Timestamp

francoisternois
Path Finder

alt text
Thank again for your quick answer. Here is an example of what I'm facing. And I've no props.conf for this sourcetype. Maybe I should configure one indicating that it's cooked data ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

On search head can you please configure below props.conf

[yoursourcetype]
KV_MODE = none
0 Karma

francoisternois
Path Finder

That's perfect ! Thank you !

0 Karma

francoisternois
Path Finder

Thank you ! It works very well ! But...
Now I have multivalue fields 😕
I guess both extracted on HF and indexer. Any ideas ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Only first full instance of splunk is parsing the data so in your case only HF is parsing data. Can you please post your multivalue field sample data and what problem are you facing with it ? Also I'll suggest to remove config from Indexer if HF is parsing data.

EDIT: It maybe due to config on Indexer, so first remove config from Indexer and then check.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...