Getting Data In
Highlighted

how to specify props.conf for EPOCH timestamp in JSON object

Path Finder

I've events coming in JSON format with first part of the JSON data as EPOCHSTARTTIME=8797994058574 ...the events are sent on Universal forwarder on some port that absorbs these events.

Question

  1. How do I specify the props.conf to recognize this to be the timestamp for the event (so that Splunk does not add index time as timestamp)
0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

Are you sure that's an epoch timestamp? I'd expect ten digits starting with a one for second precision, or thirteen digits also starting with a one for millisecond precision.

https://en.wikipedia.org/wiki/Epoch_time

0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

Path Finder

EPOCHSTARTTIME=1434209641

0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

Here are some props.conf settings. These would go in a local/props.conf on your indexers or heavy forwarders:

[mysourcetype]
TIME_PREFIX = EPOCH_START_TIME=
TIME_FORMAT = %s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14

Thanks,

Dave

Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

This would map the example timestamp to 1997 - if that's the correct interpretation you will also need to set this for mysourcetype:

MAX_DAYS_AGO = 10000

By default, Splunk will not accept timestamps older than 2000 days and instead considers them a misconfigured timestamp recognition.

Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

Esteemed Legend

The problem is that your timestamp is epoch but it is in milliseconds, not seconds, which is why it looks funny and Splunk may not understand. It could not the last time I tried it (which was a long time ago) so I pre-processed it by dividing by 1000.

0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

Splunk has no problem whatsoever recognizing millisecond-precision epoch timestamps.

This was indexed using a non-existing sourcetype, ie using Splunk's default settings:

alt text

Additionally, I doubt the question contains a millisecond-precision epoch timestamp - the example given would map to the year 2248. Not impossible, but quite unlikely indeed.

Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

Path Finder

Martin - quite correct interpretation...EPOCHSTARTTIME= 1434209641 would be actual the value. (sample data)

Still, Splunk did not recognize. When I removed the TIME_PREFIX from props.conf, it did pick it up for one of the inputs.

Very strange behavior.

0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

Splunk does support it, and has since at least v4. See my answer. If you specify your TIME_FORMAT to include them, then you will get them, using %s (time in seconds) + %

0 Karma
Highlighted

Re: how to specify props.conf for EPOCH timestamp in JSON object

SplunkTrust
SplunkTrust

Do post an entire event along with the current configuration you use for that sourcetype.
While Splunk will recognize epoch timestamps out of the box with zero configuration, some defaults may trip you. For example, it'll only look at the first 150 characters of each event by default.

0 Karma