hex encoded unix timestamp?

[maverick]

Can Splunk be configured to index my events (below) that have a hex encoded unix timestamp?

4c36117c maverick aaaaa anykey TRUE /Applications/splunk/etc/apps/search/local/blah.txt zzzzz
4c361184 maverick bbbbb allkey TRUE /Applications/splunk/etc/apps/search/local/blam.txt yyyyy

[mitch · Accepted solution]

If you want to have it do time extraction at index-time it IS possible, but only if you're willing to set up custom a custom datetime.xml file. The easier-to-use TIME_FORMAT setting doesn't have any way to specify hexadecimal numbers, so you have to get down and dirty with some XML to make it happen. Also, you'll need to be running Splunk 4.1.4 — earlier versions had more limited support for hex epoch values.

You'll need a datetime.xml file that make a regex to match the timestamp and extract it as "hexepoch". Lets call it $SPLUNK_HOME/etc/my-hex-epoch.xml:

       <datetime>
         <define name="_hexepoch" extract="hexepoch">
           <text><![CDATA[([\da-fA-F]{8})]]></text>
         </define>
         <timePatterns>
           <use name="_hexepoch"/>
         </timePatterns>
         <datePatterns>
         </datePatterns>
       </datetime>

Then in props.conf you'd configure the sourcetype as:

  DATETIME_CONFIG = /etc/my-hex-epoch.xml
  MAX_TIMESTAMP_LOOKAHEAD = 8

[beaunewcomb]

I'm using literally a copy and paste of above and Splunk isn't picking up the hex timestamp. Anyone run into this?

[maverick]

Perfect! Thanks, Mitch.

[maverick]

Not sure if I have it correct, so please tell me, but I did manage to run the following search to test out what you were saying and it still does not seem to work:

---

| eval convertedhex = 1278611836 | rex "^(?\S+?)\smaverick" | eval astrptime = strptime(atime, "%s") | convert ctime(convertedhex) ctime(astrptime)

---

The results are:

---

convertedhex=07/08/2010 12:57:16

astrptime=12/31/1969 18:00:04.000000

---

Only when I convert the hexadecimal to decimal first like the above, it appears correct.

I also tried using %+ and %S for the strptime part, but I receive the same results above.

[gkanapathy]

Yes it can. It would probably be a good idea to use a TIME_PREFIX and TIME_FORMAT for it though (%s). And a MAX_TIMESTAMP_LOOKAHEAD.

[maverick]

because its hexadecimal now and needs to be converted to unix timestamp, which looks like it's just the seconds only, will TIME_FORMAT settings actually work?