Getting Data In

hex encoded unix timestamp?

maverick
Splunk Employee
Splunk Employee

Can Splunk be configured to index my events (below) that have a hex encoded unix timestamp?

4c36117c maverick aaaaa anykey TRUE /Applications/splunk/etc/apps/search/local/blah.txt zzzzz
4c361184 maverick bbbbb allkey TRUE /Applications/splunk/etc/apps/search/local/blam.txt yyyyy

1 Solution

mitch
Explorer

If you want to have it do time extraction at index-time it IS possible, but only if you're willing to set up custom a custom datetime.xml file. The easier-to-use TIME_FORMAT setting doesn't have any way to specify hexadecimal numbers, so you have to get down and dirty with some XML to make it happen. Also, you'll need to be running Splunk 4.1.4 — earlier versions had more limited support for hex epoch values.

You'll need a datetime.xml file that make a regex to match the timestamp and extract it as "hexepoch". Lets call it $SPLUNK_HOME/etc/my-hex-epoch.xml:

       <datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[([\da-fA-F]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
       </datetime>

Then in props.conf you'd configure the sourcetype as:

  DATETIME_CONFIG = /etc/my-hex-epoch.xml
  MAX_TIMESTAMP_LOOKAHEAD = 8

View solution in original post

mitch
Explorer

If you want to have it do time extraction at index-time it IS possible, but only if you're willing to set up custom a custom datetime.xml file. The easier-to-use TIME_FORMAT setting doesn't have any way to specify hexadecimal numbers, so you have to get down and dirty with some XML to make it happen. Also, you'll need to be running Splunk 4.1.4 — earlier versions had more limited support for hex epoch values.

You'll need a datetime.xml file that make a regex to match the timestamp and extract it as "hexepoch". Lets call it $SPLUNK_HOME/etc/my-hex-epoch.xml:

       <datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[([\da-fA-F]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
       </datetime>

Then in props.conf you'd configure the sourcetype as:

  DATETIME_CONFIG = /etc/my-hex-epoch.xml
  MAX_TIMESTAMP_LOOKAHEAD = 8

beaunewcomb
Communicator

I'm using literally a copy and paste of above and Splunk isn't picking up the hex timestamp. Anyone run into this?

0 Karma

maverick
Splunk Employee
Splunk Employee

Perfect! Thanks, Mitch.

0 Karma

maverick
Splunk Employee
Splunk Employee

Not sure if I have it correct, so please tell me, but I did manage to run the following search to test out what you were saying and it still does not seem to work:


| eval convertedhex = 1278611836 | rex "^(?\S+?)\smaverick" | eval astrptime = strptime(atime, "%s") | convert ctime(convertedhex) ctime(astrptime)


The results are:


convertedhex=07/08/2010 12:57:16

astrptime=12/31/1969 18:00:04.000000


Only when I convert the hexadecimal to decimal first like the above, it appears correct.

I also tried using %+ and %S for the strptime part, but I receive the same results above.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Yes it can. It would probably be a good idea to use a TIME_PREFIX and TIME_FORMAT for it though (%s). And a MAX_TIMESTAMP_LOOKAHEAD.

0 Karma

maverick
Splunk Employee
Splunk Employee

because its hexadecimal now and needs to be converted to unix timestamp, which looks like it's just the seconds only, will TIME_FORMAT settings actually work?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...