Splunk Search

How to convert epoch time with milliseconds into splunk at indexing time

vrmandadi
Builder

I have a file that I am monitoring has time in epoch format milliseconds .What setting should be placed in the props.conf to convert it to human readable

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

You don't want to convert timestamps to human-readable format at index time because a human is not reading the timestamp at index time. Use TIME_FORMAT = %s%3N to tell Splunk the timestamp is in epoch form with milliseconds. Remember to set TIME_PREFIX properly.

Do the conversion to human-readable format at search time. Do so using fieldformat as late as possible in the query.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

manjunathmeti
Champion

Use INGEST_EVAL in transforms.conf on indexers:

props.conf

[mysourcetype]
TRANSFORMS = myeval

transforms.conf

[myeval]
INGEST_EVAL = human_readable_field = strftime(epoch_field_from_data, "%m-%d-%Y %H:%M:%S.%3N")

And on search heads add this field in fields.conf so that users can search this field.
fields.conf

[human_readable_field]
INDEXED = True
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You don't want to convert timestamps to human-readable format at index time because a human is not reading the timestamp at index time. Use TIME_FORMAT = %s%3N to tell Splunk the timestamp is in epoch form with milliseconds. Remember to set TIME_PREFIX properly.

Do the conversion to human-readable format at search time. Do so using fieldformat as late as possible in the query.

---
If this reply helps you, Karma would be appreciated.
0 Karma

HattrickNZ
Motivator

%s is for 10 digits epoch? I have a 9 digit epoch and it seems the %s is not bsorking forb me?

0 Karma

inventsekar
SplunkTrust
SplunkTrust

@HattrickNZ if you are having a 9 digit epoch means, thats timestamp value is a very old timestamp(before Sun Sep 9 01:46:39 2001 UTC)(but ideally it should work fine, something wrong.. pls provide us your search query)

Question

Will UNIX's epoch time change from 9 to 10 digits affect Gentran? (SCI8237)

Answer

No. epoch time is how time is kept track of internally in UNIX. It's seconds, counting upward from January 1st, 1970. This number hit 1 million (1,000,000) in March of 1973, and will hit one billion (1,000,000,000) on Sun Sep 9 01:46:39 2001 UTC. This change, from a number which can be represented in 9 decimal digits to a 10-digit number, is not expected to cause any problems for UNIX systems.

The reason is that this value is not stored as decimal digits. Instead, it is stored as an integer value (a 32-bit binary variable) which can be used safely until the year 2038, when the epoch date goes back to 0. The uses in UNIX of a decimal format for the "seconds time" value are primarily in portable file formats, such as tar, cpio, and ar. These formats have always supported at least eleven decimal (or octal in some cases) digits, easily handling UNIX's one-billionth "birthday".

 

https://www.ibm.com/support/pages/will-unixs-epoch-time-change-9-10-digits-affect-gentran-sci8237-st...

 

0 Karma

vrmandadi
Builder

Thank you @richgalloway .What time format do I need to set for events which have Mar 25, 21:43 UTC as timestamp

0 Karma

richgalloway
SplunkTrust
SplunkTrust

%b %d, %H:%M:%S %Z. See the "Date and time format variables" section of the Search Reference manual.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vrmandadi
Builder

Thank You.

0 Karma
Get Updates on the Splunk Community!

New Case Study Shows the Value of Partnering with Splunk Academic Alliance

The University of Nevada, Las Vegas (UNLV) is another premier research institution helping to shape the next ...

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...