topic Re: how to specify props.conf for EPOCH timestamp in JSON object in Getting Data In

how to specify props.conf for EPOCH timestamp in JSON object

ronak — Mon, 28 Sep 2020 20:15:57 GMT

I've events coming in JSON format with first part of the JSON data as EPOCH_START_TIME=8797994058574 ...the events are sent on Universal forwarder on some port that absorbs these events.

Question

How do I specify the props.conf to recognize this to be the timestamp for the event (so that Splunk does not add index time as timestamp)

Re: how to specify props.conf for EPOCH timestamp in JSON object

martin_mueller — Sun, 14 Jun 2015 08:45:51 GMT

Are you sure that's an epoch timestamp? I'd expect ten digits starting with a one for second precision, or thirteen digits also starting with a one for millisecond precision.

https://en.wikipedia.org/wiki/Epoch_time

Re: how to specify props.conf for EPOCH timestamp in JSON object

dshpritz — Sun, 14 Jun 2015 13:11:12 GMT

Here are some props.conf settings. These would go in a local/props.conf on your indexers or heavy forwarders:

[mysourcetype]
TIME_PREFIX = EPOCH_START_TIME=
TIME_FORMAT = %s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14

Thanks,

Dave

Re: how to specify props.conf for EPOCH timestamp in JSON object

woodcock — Sun, 14 Jun 2015 14:10:31 GMT

The problem is that your timestamp is epoch but it is in milliseconds, not seconds, which is why it looks funny and Splunk may not understand. It could not the last time I tried it (which was a long time ago) so I pre-processed it by dividing by 1000.

Re: how to specify props.conf for EPOCH timestamp in JSON object

martin_mueller — Sun, 14 Jun 2015 14:32:48 GMT

Splunk has no problem whatsoever recognizing millisecond-precision epoch timestamps.

This was indexed using a non-existing sourcetype, ie using Splunk's default settings:

Additionally, I doubt the question contains a millisecond-precision epoch timestamp - the example given would map to the year 2248. Not impossible, but quite unlikely indeed.

Re: how to specify props.conf for EPOCH timestamp in JSON object

dshpritz — Sun, 14 Jun 2015 14:37:06 GMT

Splunk does support it, and has since at least v4. See my answer. If you specify your TIME_FORMAT to include them, then you will get them, using %s (time in seconds) + %

Re: how to specify props.conf for EPOCH timestamp in JSON object

martin_mueller — Sun, 14 Jun 2015 14:50:53 GMT

This would map the example timestamp to 1997 - if that's the correct interpretation you will also need to set this for mysourcetype:

MAX_DAYS_AGO = 10000

By default, Splunk will not accept timestamps older than 2000 days and instead considers them a misconfigured timestamp recognition.

Re: how to specify props.conf for EPOCH timestamp in JSON object

ronak — Mon, 28 Sep 2020 20:15:59 GMT

EPOCH_START_TIME=1434209641

Re: how to specify props.conf for EPOCH timestamp in JSON object

ronak — Mon, 28 Sep 2020 20:16:02 GMT

Martin - quite correct interpretation...EPOCH_START_TIME= 1434209641 would be actual the value. (sample data)

Still, Splunk did not recognize. When I removed the TIME_PREFIX from props.conf, it did pick it up for one of the inputs.

Very strange behavior.

Re: how to specify props.conf for EPOCH timestamp in JSON object

martin_mueller — Sun, 14 Jun 2015 16:03:28 GMT

Do post an entire event along with the current configuration you use for that sourcetype.
While Splunk will recognize epoch timestamps out of the box with zero configuration, some defaults may trip you. For example, it'll only look at the first 150 characters of each event by default.