Getting Data In

Extract timestamp from two json fields

alec_stan
Explorer

I need to extract timestamp from a JSON log where date and time are on two separate fields. Example below: 

 

{  "Date": 240315, "EMVFallback": false, "FunctionCode": 80, "Time": 154915 }

 

Date here is equivalent of 2024-March-15 and the time is 15:49:15 pm.

I am struggling to find a way to extract timestamp using props.conf. May you please assist. 

Labels (2)
0 Karma

tscroggins
Influencer

Hi @alec_stan,

You can extract the timestamp using INGEST_EVAL in transforms.conf referenced by a TRANSFORMS setting in props.conf.

If your source type has INDEXED_EXTRACTIONS = json, you can reference the Date and Time fields directly in your INGEST_EVAL expression; otherwise, you can use JSON eval functions to extract the Date and Time values from _raw.

### with INDEXED_EXTRACTIONS

# props.conf

[alec_stan_json]
INDEXED_EXTRACTIONS = json
TRANSFORMS-alec_stan_json_time = alec_stan_json_time

# transforms.conf

[alec_stan_json_time]
INGEST_EVAL = _time:=strptime(tostring(Date).tostring(Time), "%y%m%d%H%M%S")

### without INDEXED_EXTRACTIONS

# props.conf

[alec_stan_json]
TRANSFORMS-alec_stan_json_time = alec_stan_json_time

# transforms.conf

[alec_stan_json_time]
INGEST_EVAL = _time:=strptime(tostring(json_extract(json(_raw), "Date")).tostring(json_extract(json(_raw), "Time")), "%y%m%d%H%M%S")

If the event time zone differs from the receiver time zone, add a time zone string (%Z) or offset (%z) to the eval expression:

[alec_stan_json_time]
INGEST_EVAL = _time:=strptime(tostring(Date).tostring(Time)."EDT", "%y%m%d%H%M%S%Z")

In a typical environment, deploy props.conf to universal forwarders and props.conf and transforms.conf to receivers (heavy forwarders and indexers).

If you haven't already, you should add SHOULD_LINEMERGE, LINE_BREAKER, etc. settings to props.conf to correctly break your input into events. You can also set DATETIME_CONFIG = CURRENT or DATETIME_CONFIG = NONE to help Splunk skip automatic timestamp extraction logic since you'll be extracting the timestamp using INGEST_EVAL.

 

Get Updates on the Splunk Community!

AppDynamics Summer Webinars

This summer, our mighty AppDynamics team is cooking up some delicious content on YouTube Live to satiate your ...

SOCin’ it to you at Splunk University

Splunk University is expanding its instructor-led learning portfolio with dedicated Security tracks at .conf25 ...

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...