topic Re: Pull Date from Filename in Splunk Search

Pull Date from Filename

adrianduff — Fri, 11 Nov 2016 07:10:36 GMT

So I have some logs that are in the following format:

Filename: 16061601rw.dat

Each line has a time stamp, but it doesn't include the date. I've checked a ton of answers about getting date from the filename but i cant get it to work properly. When it is being ingested into splunk it looks like this:

It is probably worth mentioning that where it starts stamping them as 15/06/2016, it has defaulted back to the last logs date.

Here is my attempted edit of the datetime.xml:

<text><![CDATA[source::.*?((\d{2})(\d{2})(\d{2}))*.dat]]></text>

and I have a custom data type in props.conf that I am trying to use to get it to correctly date the entries:

[custom_type]
DATETIME_CONFIG = /Applications/Splunk/etc/datetime.xml
SHOULD_LINEMERGE = FALSE

Can anyone help with this? I think it is most likely an issue with the regex but I'm not sure

Re: Pull Date from Filename

lguinn2 — Sun, 13 Nov 2016 14:33:25 GMT

FIrst, I would never do this by editing datetime.xml if there were any other way. But that is not your current problem.

Here is the problem: your props.conf is not being used.
I can tell because props.conf specifies that the input is one line per event, but that is not at all what Splunk is doing.
Where is the inputs.conf that collects this data? What kind of forwarder are you using? Where is this props.conf located? Does the sourcetype that you assigned in inputs.conf match the sourcetype you specified in props.conf?

Re: Pull Date from Filename

niketn — Tue, 29 Sep 2020 11:48:21 GMT

Although your props.conf has SHOULD_LINEMERGE as false, I am not sure why the screenshot you provided has mutiple events merged together. If every event in your log does not have end line character then you should also consider using BREAK_ONLY_BEFORE or BREAK_ONLY_AFTER parameters for props.conf.

Nevertheless, Splunk has several inbuilt algorithms with precedence in order to identify timestamp of each event. If every algorithm fails, it resorts to File Modified time. In your screenshot I can see that even filename is having same date as that logged inside the file. So if you edit Timestamp identification of props.conf, you might be able to tell Splunk to always get date from the file modified date and time from the event timestamp. In your case you may achieve this by setting MAX_TIMESTAMP_LOOKAHEAD=8 which overlooks the event Date: 16-06-2016 00:00:33 to extract datetime of the event.

[custom_type]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=AUTO
MAX_TIMESTAMP_LOOKAHEAD=8
disabled=false
invalid_cause=binary
is_valid=false

PS: If possible you should consider as to how to get the logs written with Date-Time stamp for each event rather than just time, as Time of event is the most crucial information for proper indexing of data.

Above method might fail when date rolls over from one to other around midnight.