Splunk Search

Timestamp Problem props.conf

Motivator

Hey,

I'm having difficulty getting my Splunk instance to extract the part of the timestamp that I want Splunk to set as my timestamp under _time field.

For example, I have the following timestamp entry in a log file:

8976 31/08/2010 22:55:00 Load Tue 22:55:00 Wed 00:00:57

I want splunk to use 31/08/2010 with 00:00:57 as my timestamp and ignore the rest of it. How can I configure props.conf to enable this?

I believe the right regex syntax for TIME_PREFIX may help.

Thanks in advance for your help.


LATEST as of Friday 6pm

I have written the following regex which should capture the specific date and time that I want in my timestamp as specified above:

(\d{2}\/\d{2}\/\d{4}) \d{2}\:\d{2}\:\d{2} \w{4} \w{3} \d{2}\:\d{2}\:\d{2} \w{3} (\d{2}\:\d{2}\:\d{2})

However, where would I put this regex to get Splunk to make it work?

Tags (3)
1 Solution

Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

View solution in original post

Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

View solution in original post

Motivator

Hey gkanapathy, I tested your custom file and it works! I get the timestamp I require. Thanks a lot for your help. Very much appreciated. I 'owe' you a drink 🙂

0 Karma

Motivator

I will have a look into this on Monday. Thanks for you help. I don't mean to be cheeky but if you could give it a test, I would appreciate that a lot. Could you have a look at the 'LATEST as of Friday 6pm' section in my edited question above and let me know what you think. Thanks

0 Karma

Splunk Employee
Splunk Employee

You should review this web page:

http://www.splunk.com/base/Documentation/latest/Admin/Configurepositionaltimestampextraction

The setting you probably want to use is the TIME_FORMAT:

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date. 
* For more information on strptime see `man strptime` or "Configure timestamp recognition" in the Splunk Admin Manual.
* This method of date extraction does not support in-event timezones. 
* TIME_FORMAT starts reading after the TIME_PREFIX. 
* For good results, the <strptime-style format> should describe the day of the year and the time of day.
* Defaults to empty.

For examples, you can reference the $SPLUNK_HOME/etc/system/default/props.conf settings for various sourcetypes.

0 Karma

Motivator

Thanks for the link Simeon

0 Karma