Getting Data In
Highlighted

Timestamp from file name

Motivator

I have file names like this "Patch-Data_2-1-2012.csv"

How do I use the date in the file name for the datestamp for all the lines/events in the file

This is a one shot import

Tags (2)
Highlighted

Re: Timestamp from file name

Splunk Employee
Splunk Employee

Here's what you need to do :

  • Write your own regular expression which will extract the month, the day and the year from your event source each in its own capture group.
  • Include your regex in a custom datetime.xml file. You should use $SPLUNK_HOME/etc/datetime.xml as a reference for the format to use for your own custom datetime.xml
  • Set up DATETIME_CONFIG in props.conf so that it would use your custom datetime.xml file.

Just to get you started, here's what you'll probably want to have in your own instance of datetime.xml to extract dates from your file source :



<![CDATA[(?:^|source::).*?(0?[1-9]|1[012])-(0?[1-9]|[12]\d|3[01])-(20\d\d|19\d\d|[901]\d(?!\d))\.csv]]>



(...)


(...)
<use name="sourcedate"/>

The easy way to go about this would be to copy $SPLUNK_HOME/etc/datetime.xml to your custom app and then add the regular expression definition at the end of the existing ones, and reference it under as shown above.

View solution in original post

Highlighted

Re: Timestamp from file name

Builder

hi guys, I have tried replicating this same config and filename on Splunk 5.0.2 but it is not working for me

any known issues on Splunk 5 about this??

0 Karma
Highlighted

Re: Timestamp from file name

Communicator

@asimagu
Hi, I've had the same problem, couldn't get it to work at all, Splunk used the modification date instead.

I ended up with overriding _time by using EVAL in props.conf. Not the most elegant solution, but it works for now.

However, it would be nicer to do it at index time as it should, so I really hope you fint a solution that works for me as well 🙂

0 Karma
Highlighted

Re: Timestamp from file name

New Member

It seems a time is necessary at least.

"If no events in a source have a date, Splunk Enterprise tries to find one in the source name or file name. (This requires that the events have a time, even though they don't have a date.)"

0 Karma
Highlighted

Re: Timestamp from file name

New Member

props.conf:::::
[mysourcetype]
TRANSFORMS-changetime= changetime

transforms.conf:::::
[changetime]
INGEST
EVAL = _time = strptime ( replace(source,".+(\d+-\d+-\d+).+", "\1"), "%H-%M-%S")

0 Karma