Splunk Search

Timestamp Problem props.conf

Ant1D
Motivator

Hey,

I'm having difficulty getting my Splunk instance to extract the part of the timestamp that I want Splunk to set as my timestamp under _time field.

For example, I have the following timestamp entry in a log file:

8976 31/08/2010 22:55:00 Load Tue 22:55:00 Wed 00:00:57

I want splunk to use 31/08/2010 with 00:00:57 as my timestamp and ignore the rest of it. How can I configure props.conf to enable this?

I believe the right regex syntax for TIME_PREFIX may help.

Thanks in advance for your help.


LATEST as of Friday 6pm

I have written the following regex which should capture the specific date and time that I want in my timestamp as specified above:

(\d{2}\/\d{2}\/\d{4}) \d{2}\:\d{2}\:\d{2} \w{4} \w{3} \d{2}\:\d{2}\:\d{2} \w{3} (\d{2}\:\d{2}\:\d{2})

However, where would I put this regex to get Splunk to make it work?

Tags (3)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

This will be hard if date and time are that widely separated, with things that look the same in between. You probably can't do it with TIME_PREFIX and TIME_FORMAT. Instead, you're probably best off using a custom datetime.xml config file. So, in props.conf, your sourcetype should specify:

DATETIME_CONFIG = /etc/apps/myapp/local/mycustomdatetime.xml

And then create the file (in $SPLUNK_HOME/etc/apps/myapp/local/mycustomdatetime.xml):

<datetime>
   <define name="_mydatetimeformat" extract="day, month, year, hour, minute, second">
       <text><![CDATA[^\S+\s+(\d+)/(\d+)/(\d+)\s+(?:\S+\s+){5}(\d+):(\d+):(\d+)]]></text>
   </define>
   <timePatterns>
       <use name="_mydatetimeformat"/>
   </timePatterns>
   <datePatterns>
       <use name="_mydatetimeformat"/>
   </datePatterns> 
</datetime>

I haven't really tested that regex, but basically, each capture group matches up in order with the fields that are listed in the "extract" attribute.

Ant1D
Motivator

Hey gkanapathy, I tested your custom file and it works! I get the timestamp I require. Thanks a lot for your help. Very much appreciated. I 'owe' you a drink 🙂

0 Karma

Ant1D
Motivator

I will have a look into this on Monday. Thanks for you help. I don't mean to be cheeky but if you could give it a test, I would appreciate that a lot. Could you have a look at the 'LATEST as of Friday 6pm' section in my edited question above and let me know what you think. Thanks

0 Karma

Simeon
Splunk Employee
Splunk Employee

You should review this web page:

http://www.splunk.com/base/Documentation/latest/Admin/Configurepositionaltimestampextraction

The setting you probably want to use is the TIME_FORMAT:

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date. 
* For more information on strptime see `man strptime` or "Configure timestamp recognition" in the Splunk Admin Manual.
* This method of date extraction does not support in-event timezones. 
* TIME_FORMAT starts reading after the TIME_PREFIX. 
* For good results, the <strptime-style format> should describe the day of the year and the time of day.
* Defaults to empty.

For examples, you can reference the $SPLUNK_HOME/etc/system/default/props.conf settings for various sourcetypes.

0 Karma

Ant1D
Motivator

Thanks for the link Simeon

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...