topic Re: Timestamp Problem props.conf in Splunk Search

Timestamp Problem props.conf

Ant1D — Fri, 03 Sep 2010 22:30:19 GMT

Hey,

I'm having difficulty getting my Splunk instance to extract the part of the timestamp that I want Splunk to set as my timestamp under _time field.

For example, I have the following timestamp entry in a log file:

8976 31/08/2010 22:55:00 Load Tue 22:55:00 Wed 00:00:57

I want splunk to use 31/08/2010 with 00:00:57 as my timestamp and ignore the rest of it. How can I configure props.conf to enable this?

I believe the right regex syntax for TIME_PREFIX may help.

Thanks in advance for your help.

LATEST as of Friday 6pm

I have written the following regex which should capture the specific date and time that I want in my timestamp as specified above:

(\d{2}\/\d{2}\/\d{4}) \d{2}\:\d{2}\:\d{2} \w{4} \w{3} \d{2}\:\d{2}\:\d{2} \w{3} (\d{2}\:\d{2}\:\d{2})

However, where would I put this regex to get Splunk to make it work?

Re: Timestamp Problem props.conf

Simeon — Fri, 03 Sep 2010 22:42:24 GMT

You should review this web page:

http://www.splunk.com/base/Documentation/latest/Admin/Configurepositionaltimestampextraction

The setting you probably want to use is the TIME_FORMAT:

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date. 
* For more information on strptime see `man strptime` or "Configure timestamp recognition" in the Splunk Admin Manual.
* This method of date extraction does not support in-event timezones. 
* TIME_FORMAT starts reading after the TIME_PREFIX. 
* For good results, the <strptime-style format> should describe the day of the year and the time of day.
* Defaults to empty.

For examples, you can reference the $SPLUNK_HOME/etc/system/default/props.conf settings for various sourcetypes.

Re: Timestamp Problem props.conf

gkanapathy — Fri, 03 Sep 2010 23:57:19 GMT

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

Re: Timestamp Problem props.conf

Ant1D — Sat, 04 Sep 2010 00:12:11 GMT

I will have a look into this on Monday. Thanks for you help. I don't mean to be cheeky but if you could give it a test, I would appreciate that a lot. Could you have a look at the 'LATEST as of Friday 6pm' section in my edited question above and let me know what you think. Thanks

Re: Timestamp Problem props.conf

Ant1D — Mon, 06 Sep 2010 20:27:43 GMT

Hey gkanapathy, I tested your custom file and it works! I get the timestamp I require. Thanks a lot for your help. Very much appreciated. I 'owe' you a drink 🙂

Re: Timestamp Problem props.conf

Ant1D — Mon, 06 Sep 2010 20:28:13 GMT

Thanks for the link Simeon