How do you extract a timestamp from a filename?

[AKG1_old1]

Hello,

I am looking to extract a timestamp from a filename.

Example:
jstack_dell730srv_18_12_07_15_28_44.log 

Syntax:
jstack_dell730srv_(year)_(month)_(day)_(hour)_(minute)_(second).log

I am looking to update _time during indexing.

I have tried using DATETIME_CONFIG but no luck.

props.conf
[jstack]
DATETIME_CONFIG = /etc/apps/myapp/local/datetime.xml
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

/etc/apps/myapp/local/datetime.xml

    <define name="_mydatetime" extract="month, day, hour, minute, second">
    <text><![CDATA[source::.*_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d).log]]></text>
 </define>
 <timePatterns>
    <use name="_mydatetime"/>
 </timePatterns>
 <datePatterns>
    <use name="_mydatetime"/>
 </datePatterns> 

Also I have tried this in props.conf but not working.

[jstack]
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

[mthomas_splunk]

This is possible in Splunk Enterprise 7.2, making use of the new ingest-time eval. Full documentation is at https://docs.splunk.com/Documentation/Splunk/latest/Data/IngestEval.

Example

File Name: Log_I15_13092018183001.txt
File Name Format: Log_I15_%d%m%Y%H%M%S.txt

props.conf

[mysourcetype]
TRANSFORMS=timestampeval

transforms.conf

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"Log_I15_%d%m%Y%H%M%S.txt")

This takes the "source" metadata value (which is the path and file name), removes the path, then extracts the date and time from the filename.

All events in the file will have the same _time when imported.

[mah]

Hi @mthomas_splunk 

In my case, how can I index multiple file with only one INGEST-EVAL ?

For instance : 

prod-1-%d%m%Y%H%M%S.txt

prod-2-%d%m%Y%H%M%S.txt

prod-3-%d%m%Y%H%M%S.txt

I tried this :

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"prod-.-%d%m%Y%H%M%S.txt")

But doesn't work... 

[burwell]

Hello @mah 

I suggest you submit a new question as this is a different question. Thanks!

[FrankVl]

If that timestamp in the filename matches the modification time of the file, you could also set DATETIME_CONFIG = none. Splunk will then take the file mod time as the timestamp.

Using props extract and eval, your attempt does not have a correct EXTRACT syntax. You need a named capture group.

Try like this

EXTRACT-Time = _(?<Time>\d+_\d+_\d+_\d+_\d+).log$ in source
EVAL-_time = strptime(Time,"%m_%d_%H_%M_%S")

As far as I know, Splunk automatically uses the current year when you don't specify one. If not, you may need to get a bit more creative.
But note: this only sets _time to the timestamp in the filename at search time, not at index time, so not sure if that really is what you want.

If you run Splunk 7.2 or newer, you could take a look at the INGEST_EVAL construct in transforms.conf. That way you could do the _time evaluation at index time.
props.conf:

TRANSFORM-Time = get-time-from-source

transforms.conf

[get-time-from-source]
INGEST_EVAL = _time=strptime(replace(source,".+_(\d+_\d+_\d+_\d+_\d+).log$","\1"),"%m_%d_%H_%M_%S")

[amsinha_splunk]

Hello Frank !! 
This logic is working fine using the direct test input on the indexer (Using Add data), but something is different when using the UF. Could you please help me to figure out how they ingest the timestamp from the filename on the UF?

[ssadanala1]

Hope this helps

https://answers.splunk.com/answers/320978/how-to-extract-the-timestamp-from-a-filename-at-in.html

[AKG1_old1]

Thanks I refered this post initially looks like I am doing the same thing but its not working.

[Vijeta]

You can extract date or timestamp from file using Transforms , write a regex to extract the timestamp , you can try it from UI as well (Field Transformation)and then define in props.conf(Field Extractions)

[AKG1_old1]

@Vijeta : Added year manaually in filename. I have tried this but its not working. I have added this in props.conf. In this case I am getting blank _time. Do I miss something ?

props.conf
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")

[macadminrohit]

can he extract just the date,month,hour,minute,second from the available file name and use now() to get the current year?