topic Re: hex encoded unix timestamp? in Getting Data In

hex encoded unix timestamp?

maverick — Wed, 21 Jul 2010 09:43:16 GMT

Can Splunk be configured to index my events (below) that have a hex encoded unix timestamp?

4c36117c maverick aaaaa anykey TRUE /Applications/splunk/etc/apps/search/local/blah.txt zzzzz
4c361184 maverick bbbbb allkey TRUE /Applications/splunk/etc/apps/search/local/blam.txt yyyyy

Re: hex encoded unix timestamp?

gkanapathy — Wed, 21 Jul 2010 10:02:02 GMT

Yes it can. It would probably be a good idea to use a TIME_PREFIX and TIME_FORMAT for it though (%s). And a MAX_TIMESTAMP_LOOKAHEAD.

Re: hex encoded unix timestamp?

maverick — Wed, 21 Jul 2010 10:17:27 GMT

because its hexadecimal now and needs to be converted to unix timestamp, which looks like it's just the seconds only, will TIME_FORMAT settings actually work?

Re: hex encoded unix timestamp?

maverick — Wed, 21 Jul 2010 21:55:26 GMT

Not sure if I have it correct, so please tell me, but I did manage to run the following search to test out what you were saying and it still does not seem to work:

| eval convertedhex = 1278611836 | rex "^(?\S+?)\smaverick" | eval astrptime = strptime(atime, "%s") | convert ctime(convertedhex) ctime(astrptime)

The results are:

convertedhex=07/08/2010 12:57:16

astrptime=12/31/1969 18:00:04.000000

Only when I convert the hexadecimal to decimal first like the above, it appears correct.

I also tried using %+ and %S for the strptime part, but I receive the same results above.

Re: hex encoded unix timestamp?

mitch — Thu, 22 Jul 2010 14:44:42 GMT

If you want to have it do time extraction at index-time it IS possible, but only if you're willing to set up custom a custom datetime.xml file. The easier-to-use TIME_FORMAT setting doesn't have any way to specify hexadecimal numbers, so you have to get down and dirty with some XML to make it happen. Also, you'll need to be running Splunk 4.1.4 — earlier versions had more limited support for hex epoch values.

You'll need a datetime.xml file that make a regex to match the timestamp and extract it as "hexepoch". Lets call it $SPLUNK_HOME/etc/my-hex-epoch.xml:

       <datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[([\da-fA-F]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
       </datetime>

Then in props.conf you'd configure the sourcetype as:

  DATETIME_CONFIG = /etc/my-hex-epoch.xml
  MAX_TIMESTAMP_LOOKAHEAD = 8

Re: hex encoded unix timestamp?

maverick — Thu, 22 Jul 2010 20:19:11 GMT

Perfect! Thanks, Mitch.

Re: hex encoded unix timestamp?

beaunewcomb — Wed, 23 Apr 2014 23:08:59 GMT

I'm using literally a copy and paste of above and Splunk isn't picking up the hex timestamp. Anyone run into this?