Getting Data In

Since March 13th 2011 GMT, Splunk no longer properly parses my epoch timestamps (seconds since 1970)

carasso
Splunk Employee
Splunk Employee

Since March 13th 2011 GMT, Splunk no longer properly parses my epoch timestamps (seconds since 1970).

For example, I am indexing the following log file :


[root@beefysup01 splunk]# ls -l /test/epoch/asterisk.log 
-rw-r--r-- 1 root root 236 Mar 15 13:24 /test/epoch/asterisk.log

Contents of the file :


1300218018|NONE|NONE|NONE|CONFIGRELOAD|
1300218018|NONE|NONE|NONE|CONFIGRELOAD|
1300218957|NONE|NONE|NONE|QUEUESTART|
1300218958|NONE|NONE|NONE|CONFIGRELOAD|
1300219007|NONE|NONE|NONE|QUEUESTART|
1300219007|NONE|NONE|NONE|CONFIGRELOAD|

We expect Splunk to assign the following time stamps to these events :


Tue Mar 15 12:40:18 PDT 2011 : 1300218018|NONE|NONE|NONE|CONFIGRELOAD|
Tue Mar 15 12:40:18 PDT 2011 : 1300218018|NONE|NONE|NONE|CONFIGRELOAD|
Tue Mar 15 12:55:57 PDT 2011 : 1300218957|NONE|NONE|NONE|QUEUESTART|
Tue Mar 15 12:55:58 PDT 2011 : 1300218958|NONE|NONE|NONE|CONFIGRELOAD|
Tue Mar 15 12:56:47 PDT 2011 : 1300219007|NONE|NONE|NONE|QUEUESTART|
Tue Mar 15 12:56:47 PDT 2011 : 1300219007|NONE|NONE|NONE|CONFIGRELOAD|

Instead, Splunk is assigning the same time stamp to all events : Tue Mar 15 13:24:00 PDT 2011 - This is the modtime of the file and not accurate in the context of each event.

Strangely, there are no DateParserVerbose errors recorded in splunkd.log.

1 Solution

carasso
Splunk Employee
Splunk Employee

In Splunk's datetime.xml, the regular expression for parsing epoch times, assumes values from 2002 through to March 13th 2011. Those values started with 10,11,12. On Sunday March 13th 2011 07:06:40 GMT, the seconds since 1970 became 1300000000, starting with 13.

This will be fixed in 4.2.1.

There are two possible fixes. The first involves editing an xml file and will resolve the problem for all data sources. The second doesn involves editing props.conf and will fix things for whatever source/sourcetype/host you change.


1) The fix is to modify $SPLUNK_HOME/etc/datetime.xml, around line 200. Change the _utcepoch regex to the following:

<define name="_utcepoch" extract="utcepoch, subsecond">
    <!-- update regex before 2017! 🙂 -->
    <text><![CDATA[((?<=^|[\s#,"=([\|{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:.?(\d{1,6}))?(?![\d(])]]></text>
</define>

Make sure to make a backup of your file first.


2) Alternatively, for your sources that use epoch time, explicitly specify a strptime format in props.conf, by using TIME_FORMAT and TIME_PREFIX fields. For example, see http://answers.splunk.com/questions/8428/how-do-i-recognize-a-time-in-epoch-seconds

View solution in original post

carasso
Splunk Employee
Splunk Employee

In Splunk's datetime.xml, the regular expression for parsing epoch times, assumes values from 2002 through to March 13th 2011. Those values started with 10,11,12. On Sunday March 13th 2011 07:06:40 GMT, the seconds since 1970 became 1300000000, starting with 13.

This will be fixed in 4.2.1.

There are two possible fixes. The first involves editing an xml file and will resolve the problem for all data sources. The second doesn involves editing props.conf and will fix things for whatever source/sourcetype/host you change.


1) The fix is to modify $SPLUNK_HOME/etc/datetime.xml, around line 200. Change the _utcepoch regex to the following:

<define name="_utcepoch" extract="utcepoch, subsecond">
    <!-- update regex before 2017! 🙂 -->
    <text><![CDATA[((?<=^|[\s#,"=([\|{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:.?(\d{1,6}))?(?![\d(])]]></text>
</define>

Make sure to make a backup of your file first.


2) Alternatively, for your sources that use epoch time, explicitly specify a strptime format in props.conf, by using TIME_FORMAT and TIME_PREFIX fields. For example, see http://answers.splunk.com/questions/8428/how-do-i-recognize-a-time-in-epoch-seconds

jplumsdaine22
Influencer

In the datetime.xml for 6.3.2 I can see the following :

<define name="_utcepoch" extract="utcepoch, subsecond">
      <!-- update regex before '2017' -->

But the regex (it is: (?:1[012345]|9)) looks good to 1599999999, or the 13th of September 2020

So hopefully we'll see an update in the next couple of years

0 Karma

acharlieh
Influencer

Not sure how you're doing epoch time conversion @jplumsdaine22 but 1599999999 is Sun, 13 Sep 2020 12:26:39 GMT.

0 Karma

jplumsdaine22
Influencer

Derp I copied the wrong line from my converter. Updated my original comment

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>