Splunk Search

Timestamp Problem props.conf

Ant1D
Motivator

Hey,

I'm having difficulty getting my Splunk instance to extract the part of the timestamp that I want Splunk to set as my timestamp under _time field.

For example, I have the following timestamp entry in a log file:

8976 31/08/2010 22:55:00 Load Tue 22:55:00 Wed 00:00:57

I want splunk to use 31/08/2010 with 00:00:57 as my timestamp and ignore the rest of it. How can I configure props.conf to enable this?

I believe the right regex syntax for TIME_PREFIX may help.

Thanks in advance for your help.


LATEST as of Friday 6pm

I have written the following regex which should capture the specific date and time that I want in my timestamp as specified above:

(\d{2}\/\d{2}\/\d{4}) \d{2}\:\d{2}\:\d{2} \w{4} \w{3} \d{2}\:\d{2}\:\d{2} \w{3} (\d{2}\:\d{2}\:\d{2})

However, where would I put this regex to get Splunk to make it work?

Tags (3)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

Ant1D
Motivator

Hey gkanapathy, I tested your custom file and it works! I get the timestamp I require. Thanks a lot for your help. Very much appreciated. I 'owe' you a drink 🙂

0 Karma

Ant1D
Motivator

I will have a look into this on Monday. Thanks for you help. I don't mean to be cheeky but if you could give it a test, I would appreciate that a lot. Could you have a look at the 'LATEST as of Friday 6pm' section in my edited question above and let me know what you think. Thanks

0 Karma

Simeon
Splunk Employee
Splunk Employee

You should review this web page:

http://www.splunk.com/base/Documentation/latest/Admin/Configurepositionaltimestampextraction

The setting you probably want to use is the TIME_FORMAT:

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date. 
* For more information on strptime see `man strptime` or "Configure timestamp recognition" in the Splunk Admin Manual.
* This method of date extraction does not support in-event timezones. 
* TIME_FORMAT starts reading after the TIME_PREFIX. 
* For good results, the <strptime-style format> should describe the day of the year and the time of day.
* Defaults to empty.

For examples, you can reference the $SPLUNK_HOME/etc/system/default/props.conf settings for various sourcetypes.

0 Karma

Ant1D
Motivator

Thanks for the link Simeon

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...