Getting Data In
Highlighted

How do you extract a timestamp from a filename?

Builder

Hello,

I am looking to extract a timestamp from a filename.

Example:
jstack_dell730srv_18_12_07_15_28_44.log 

Syntax:
jstack_dell730srv_(year)_(month)_(day)_(hour)_(minute)_(second).log

I am looking to update _time during indexing.

I have tried using DATETIME_CONFIG but no luck.

props.conf
[jstack]
DATETIMECONFIG = /etc/apps/myapp/local/datetime.xml
NO
BINARYCHECK = true
category = Custom
pulldown
type = 1

/etc/apps/myapp/local/datetime.xml

    <define name="_mydatetime" extract="month, day, hour, minute, second">
    <text><![CDATA[source::.*_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d).log]]></text>
 </define>
 <timePatterns>
    <use name="_mydatetime"/>
 </timePatterns>
 <datePatterns>
    <use name="_mydatetime"/>
 </datePatterns> 

Also I have tried this in props.conf but not working.

[jstack]
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Influencer

You can extract date or timestamp from file using Transforms , write a regex to extract the timestamp , you can try it from UI as well (Field Transformation)and then define in props.conf(Field Extractions)

0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Contributor

can he extract just the date,month,hour,minute,second from the available file name and use now() to get the current year?

0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Builder

@Vijeta : Added year manaually in filename. I have tried this but its not working. I have added this in props.conf. In this case I am getting blank _time. Do I miss something ?

props.conf
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")
0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Contributor
0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Builder

Thanks I refered this post initially looks like I am doing the same thing but its not working.

0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

Ultra Champion

If that timestamp in the filename matches the modification time of the file, you could also set DATETIME_CONFIG = none. Splunk will then take the file mod time as the timestamp.

Using props extract and eval, your attempt does not have a correct EXTRACT syntax. You need a named capture group.

Try like this

EXTRACT-Time = _(?<Time>\d+_\d+_\d+_\d+_\d+).log$ in source
EVAL-_time = strptime(Time,"%m_%d_%H_%M_%S")

As far as I know, Splunk automatically uses the current year when you don't specify one. If not, you may need to get a bit more creative.
But note: this only sets _time to the timestamp in the filename at search time, not at index time, so not sure if that really is what you want.

If you run Splunk 7.2 or newer, you could take a look at the INGEST_EVAL construct in transforms.conf. That way you could do the _time evaluation at index time.
props.conf:

TRANSFORM-Time = get-time-from-source

transforms.conf

[get-time-from-source]
INGEST_EVAL = _time=strptime(replace(source,".+_(\d+_\d+_\d+_\d+_\d+).log$","\1"),"%m_%d_%H_%M_%S")
Highlighted

Re: How do you extract a timestamp from a filename?

Explorer

This is possible in Splunk Enterprise 7.2, making use of the new ingest-time eval. Full documentation is at https://docs.splunk.com/Documentation/Splunk/latest/Data/IngestEval.

Example

File Name: LogI1513092018183001.txt
File Name Format: LogI15%d%m%Y%H%M%S.txt

props.conf

[mysourcetype]
TRANSFORMS=timestampeval

transforms.conf

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"Log_I15_%d%m%Y%H%M%S.txt")

This takes the "source" metadata value (which is the path and file name), removes the path, then extracts the date and time from the filename.

All events in the file will have the same _time when imported.

Highlighted

Re: How do you extract a timestamp from a filename?

Explorer

Hi @mthomas_splunk 

In my case, how can I index multiple file with only one INGEST-EVAL ?

For instance : 

prod-1-%d%m%Y%H%M%S.txt

prod-2-%d%m%Y%H%M%S.txt

prod-3-%d%m%Y%H%M%S.txt

I tried this :

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"prod-.-%d%m%Y%H%M%S.txt")

But doesn't work... 

Tags (1)
0 Karma
Highlighted

Re: How do you extract a timestamp from a filename?

SplunkTrust
SplunkTrust

Hello @mah 

I suggest you submit a new question as this is a different question. Thanks!

Tags (1)
0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.