topic How to convert epoch time with milliseconds into splunk at indexing time in Splunk Search

How to convert epoch time with milliseconds into splunk at indexing time

vrmandadi — Thu, 26 Mar 2020 16:26:06 GMT

I have a file that I am monitoring has time in epoch format milliseconds .What setting should be placed in the props.conf to convert it to human readable

Re: How to convert epoch time with milliseconds into splunk at indexing time

richgalloway — Thu, 26 Mar 2020 16:44:41 GMT

You don't want to convert timestamps to human-readable format at index time because a human is not reading the timestamp at index time. Use TIME_FORMAT = %s%3N to tell Splunk the timestamp is in epoch form with milliseconds. Remember to set TIME_PREFIX properly.

Do the conversion to human-readable format at search time. Do so using fieldformat as late as possible in the query.

Re: How to convert epoch time with milliseconds into splunk at indexing time

manjunathmeti — Thu, 26 Mar 2020 16:45:17 GMT

Use INGEST_EVAL in transforms.conf on indexers:

props.conf

[mysourcetype]
TRANSFORMS = myeval

transforms.conf

[myeval]
INGEST_EVAL = human_readable_field = strftime(epoch_field_from_data, "%m-%d-%Y %H:%M:%S.%3N")

And on search heads add this field in fields.conf so that users can search this field.
fields.conf

[human_readable_field]
INDEXED = True

Re: How to convert epoch time with milliseconds into splunk at indexing time

vrmandadi — Thu, 26 Mar 2020 20:22:55 GMT

Thank you @richgalloway .What time format do I need to set for events which have Mar 25, 21:43 UTC as timestamp

Re: How to convert epoch time with milliseconds into splunk at indexing time

richgalloway — Thu, 26 Mar 2020 21:04:15 GMT

%b %d, %H:%M:%S %Z. See the "Date and time format variables" section of the Search Reference manual.

Re: How to convert epoch time with milliseconds into splunk at indexing time

vrmandadi — Thu, 26 Mar 2020 21:14:06 GMT

Thank You.

Re: How to convert epoch time with milliseconds into splunk at indexing time

HattrickNZ — Sat, 03 Jul 2021 03:56:07 GMT

%s is for 10 digits epoch? I have a 9 digit epoch and it seems the %s is not bsorking forb me?

Re: How to convert epoch time with milliseconds into splunk at indexing time

inventsekar — Sun, 04 Jul 2021 08:39:54 GMT

@HattrickNZ if you are having a 9 digit epoch means, thats timestamp value is a very old timestamp(before Sun Sep 9 01:46:39 2001 UTC)(but ideally it should work fine, something wrong.. pls provide us your search query)

Question

Will UNIX's epoch time change from 9 to 10 digits affect Gentran? (SCI8237)

Answer

No. epoch time is how time is kept track of internally in UNIX. It's seconds, counting upward from January 1st, 1970. This number hit 1 million (1,000,000) in March of 1973, and will hit one billion (1,000,000,000) on Sun Sep 9 01:46:39 2001 UTC. This change, from a number which can be represented in 9 decimal digits to a 10-digit number, is not expected to cause any problems for UNIX systems.

The reason is that this value is not stored as decimal digits. Instead, it is stored as an integer value (a 32-bit binary variable) which can be used safely until the year 2038, when the epoch date goes back to 0. The uses in UNIX of a decimal format for the "seconds time" value are primarily in portable file formats, such as tar, cpio, and ar. These formats have always supported at least eleven decimal (or octal in some cases) digits, easily handling UNIX's one-billionth "birthday".

https://www.ibm.com/support/pages/will-unixs-epoch-time-change-9-10-digits-affect-gentran-sci8237-sterling-gentranserver-unix