Getting Data In

How do you extract a timestamp from a filename?

AKG1_old1
Builder

Hello,

I am looking to extract a timestamp from a filename.

Example:
jstack_dell730srv_18_12_07_15_28_44.log 

Syntax:
jstack_dell730srv_(year)_(month)_(day)_(hour)_(minute)_(second).log

I am looking to update _time during indexing.

I have tried using DATETIME_CONFIG but no luck.

props.conf
[jstack]
DATETIME_CONFIG = /etc/apps/myapp/local/datetime.xml
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1

/etc/apps/myapp/local/datetime.xml

    <define name="_mydatetime" extract="month, day, hour, minute, second">
    <text><![CDATA[source::.*_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d)_(\d\d).log]]></text>
 </define>
 <timePatterns>
    <use name="_mydatetime"/>
 </timePatterns>
 <datePatterns>
    <use name="_mydatetime"/>
 </datePatterns> 

Also I have tried this in props.conf but not working.

[jstack]
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
0 Karma

mthomas_splunk
Splunk Employee
Splunk Employee

This is possible in Splunk Enterprise 7.2, making use of the new ingest-time eval. Full documentation is at https://docs.splunk.com/Documentation/Splunk/latest/Data/IngestEval.

Example

File Name: Log_I15_13092018183001.txt
File Name Format: Log_I15_%d%m%Y%H%M%S.txt

props.conf

[mysourcetype]
TRANSFORMS=timestampeval

transforms.conf

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"Log_I15_%d%m%Y%H%M%S.txt")

This takes the "source" metadata value (which is the path and file name), removes the path, then extracts the date and time from the filename.

All events in the file will have the same _time when imported.

mah
Builder

Hi @mthomas_splunk 

In my case, how can I index multiple file with only one INGEST-EVAL ?

For instance : 

prod-1-%d%m%Y%H%M%S.txt

prod-2-%d%m%Y%H%M%S.txt

prod-3-%d%m%Y%H%M%S.txt

I tried this :

[timestampeval]
INGEST_EVAL = _time=strptime(replace(source,".*(?=/)/",""),"prod-.-%d%m%Y%H%M%S.txt")

But doesn't work... 

Tags (1)
0 Karma

burwell
SplunkTrust
SplunkTrust

Hello @mah 

I suggest you submit a new question as this is a different question. Thanks!

Tags (1)
0 Karma

FrankVl
Ultra Champion

If that timestamp in the filename matches the modification time of the file, you could also set DATETIME_CONFIG = none. Splunk will then take the file mod time as the timestamp.

Using props extract and eval, your attempt does not have a correct EXTRACT syntax. You need a named capture group.

Try like this

EXTRACT-Time = _(?<Time>\d+_\d+_\d+_\d+_\d+).log$ in source
EVAL-_time = strptime(Time,"%m_%d_%H_%M_%S")

As far as I know, Splunk automatically uses the current year when you don't specify one. If not, you may need to get a bit more creative.
But note: this only sets _time to the timestamp in the filename at search time, not at index time, so not sure if that really is what you want.

If you run Splunk 7.2 or newer, you could take a look at the INGEST_EVAL construct in transforms.conf. That way you could do the _time evaluation at index time.
props.conf:

TRANSFORM-Time = get-time-from-source

transforms.conf

[get-time-from-source]
INGEST_EVAL = _time=strptime(replace(source,".+_(\d+_\d+_\d+_\d+_\d+).log$","\1"),"%m_%d_%H_%M_%S")

amsinha_splunk
Splunk Employee
Splunk Employee

Hello Frank !! 
This logic is working fine using the direct test input on the indexer (Using Add data), but something is different when using the UF. Could you please help me to figure out how they ingest the timestamp from the filename on the UF?

0 Karma

ssadanala1
Contributor
0 Karma

AKG1_old1
Builder

Thanks I refered this post initially looks like I am doing the same thing but its not working.

0 Karma

Vijeta
Influencer

You can extract date or timestamp from file using Transforms , write a regex to extract the timestamp , you can try it from UI as well (Field Transformation)and then define in props.conf(Field Extractions)

0 Karma

AKG1_old1
Builder

@Vijeta : Added year manaually in filename. I have tried this but its not working. I have added this in props.conf. In this case I am getting blank _time. Do I miss something ?

props.conf
EXTRACT-Time = .*_(\d+)_(\d+)_(\d+)_(\d+)_(\d+)_(\d+).log$ in source
EVAL-_time = strptime(Time,"%y_%m_%d_%H_%M_%S")
0 Karma

macadminrohit
Contributor

can he extract just the date,month,hour,minute,second from the available file name and use now() to get the current year?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...