Getting Data In

How to assign custom JSON field with epoch time as the timestamp for events?

akhanVG
Path Finder

We are inputting JSON fields to splunk. One of the fields eventTime should be the event time for the index.

{
    browserType:  Mozilla/5.0 (iPhone; CPU iPhone OS 8_1_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B466 Safari/600.1.4 
    campaignLocation:  null 
    campaignName:  null 
    currentPage:  /test
    eventBy:  application 
    eventName:  pageLoad 
    eventSource:  frontend 
    eventTime:  1424822395 
    ipAddress:  127.1.1.1:45770 
    isMobile:  true 
    referrer:  http://tfdf.dfdf.com
    sessionId:  null 
    userId:  null 
}

The eventTime is in millisecond format (UTC)

This is what the JSON value looks like raw in the log file

{"browserType":"Mozilla/5.0 (Linux; Android 5.0; SM-G900V Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Mobile Safari/537.36","campaignLocation":null,"campaignName":null,"currentPage":"/dfd6","eventBy":"application","eventName":"pageLoad","eventSource":"frontend","eventTime":"1424822393","ipAddress":"192.168.1.1:58674","isMobile":true,"referrer":"http:dfsdf,"sessionId":null,"userId":null}

1 Solution

s2_splunk
Splunk Employee
Splunk Employee

OK, your complete props.conf for this sourcetype as it exists on the indexer needs to look like this (assuming every event is on its own line):

[hermes]
TIME_PREFIX=eventTime:\s+
TIME_FORMAT=%s
KV_MODE=json

If your timestamp is not in fact an epoch time, but milliseconds, try
TIME_FORMAT=%s%3N

If you can't get it to work, I suggest you use the DataPreview tool in the Splunk UI, but this is very straightforward.

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

OK, your complete props.conf for this sourcetype as it exists on the indexer needs to look like this (assuming every event is on its own line):

[hermes]
TIME_PREFIX=eventTime:\s+
TIME_FORMAT=%s
KV_MODE=json

If your timestamp is not in fact an epoch time, but milliseconds, try
TIME_FORMAT=%s%3N

If you can't get it to work, I suggest you use the DataPreview tool in the Splunk UI, but this is very straightforward.

s2_splunk
Splunk Employee
Splunk Employee

Well, I just noticed your raw json up there, so the prefix (of course) needs to be:

    TIME_PREFIX=\"eventTime\":\"

akhanVG
Path Finder

Awesome. Should have added the raw JSON.

This worked. Also I did not have it in EPOCH derp but millisecond so %s%3N worked beautiful.

s2_splunk
Splunk Employee
Splunk Employee

You had the raw message up there, I just didn't see it. Please accept the answer, if all works now.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee
TIME_PREFIX = eventTime:\s+

Docs on Timestamp Extraction

akhanVG
Path Finder

No dice. Is this supposed to go into the inputs.conf file on the forwarder or some place else?

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Line breaking and timestamp extraction happen on the indexer, so this needs to go in props.conf for your sourcetype and be available on the indexer.

Where do I configure my Splunk settings?

0 Karma

akhanVG
Path Finder

Yep - did but no dice

[rule::timestamp_hermes]
sourcetype = hermes
TIME_PREFIX = eventTime:\s+

is what I appended to the end of props.conf in /opt/splunk/etc/system/local/

Still doesn't appear to pick up the time.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

That's not what that's supposed to look like. Try this (the backslash in the RegEx is important, you had removed it):

[hermes]
TIME_PREFIX= eventTime:\s+
0 Karma

akhanVG
Path Finder

Yes that is what I have - still no good. I evne tried replacing hermes (the sourcetype) with ecom (the index name) and still no luck.

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...