Splunk Search

How to convert epoch time with milliseconds into splunk at indexing time

vrmandadi
Builder

I have a file that I am monitoring has time in epoch format milliseconds .What setting should be placed in the props.conf to convert it to human readable

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You don't want to convert timestamps to human-readable format at index time because a human is not reading the timestamp at index time. Use TIME_FORMAT = %s%3N to tell Splunk the timestamp is in epoch form with milliseconds. Remember to set TIME_PREFIX properly.

Do the conversion to human-readable format at search time. Do so using fieldformat as late as possible in the query.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Use INGEST_EVAL in transforms.conf on indexers:

props.conf

[mysourcetype]
TRANSFORMS = myeval

transforms.conf

[myeval]
INGEST_EVAL = human_readable_field = strftime(epoch_field_from_data, "%m-%d-%Y %H:%M:%S.%3N")

And on search heads add this field in fields.conf so that users can search this field.
fields.conf

[human_readable_field]
INDEXED = True
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You don't want to convert timestamps to human-readable format at index time because a human is not reading the timestamp at index time. Use TIME_FORMAT = %s%3N to tell Splunk the timestamp is in epoch form with milliseconds. Remember to set TIME_PREFIX properly.

Do the conversion to human-readable format at search time. Do so using fieldformat as late as possible in the query.

---
If this reply helps you, Karma would be appreciated.
0 Karma

HattrickNZ
Motivator

%s is for 10 digits epoch? I have a 9 digit epoch and it seems the %s is not bsorking forb me?

0 Karma

inventsekar
Ultra Champion

@HattrickNZ if you are having a 9 digit epoch means, thats timestamp value is a very old timestamp(before Sun Sep 9 01:46:39 2001 UTC)(but ideally it should work fine, something wrong.. pls provide us your search query)

Question

Will UNIX's epoch time change from 9 to 10 digits affect Gentran? (SCI8237)

Answer

No. epoch time is how time is kept track of internally in UNIX. It's seconds, counting upward from January 1st, 1970. This number hit 1 million (1,000,000) in March of 1973, and will hit one billion (1,000,000,000) on Sun Sep 9 01:46:39 2001 UTC. This change, from a number which can be represented in 9 decimal digits to a 10-digit number, is not expected to cause any problems for UNIX systems.

The reason is that this value is not stored as decimal digits. Instead, it is stored as an integer value (a 32-bit binary variable) which can be used safely until the year 2038, when the epoch date goes back to 0. The uses in UNIX of a decimal format for the "seconds time" value are primarily in portable file formats, such as tar, cpio, and ar. These formats have always supported at least eleven decimal (or octal in some cases) digits, easily handling UNIX's one-billionth "birthday".

 

https://www.ibm.com/support/pages/will-unixs-epoch-time-change-9-10-digits-affect-gentran-sci8237-st...

 

0 Karma

vrmandadi
Builder

Thank you @richgalloway .What time format do I need to set for events which have Mar 25, 21:43 UTC as timestamp

0 Karma

richgalloway
SplunkTrust
SplunkTrust

%b %d, %H:%M:%S %Z. See the "Date and time format variables" section of the Search Reference manual.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vrmandadi
Builder

Thank You.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...