topic Re: Extract data from filename in Splunk Search

Extract data from filename

Karthikeyan — Thu, 19 Aug 2021 12:47:12 GMT

Hi Experts,

I have a a job log file, that gets ingested to Splunk with naming convention "trace_08_19_2021_06_36_03_*********.txt". After trace, the format of the log file is _MM_DD_YYYY_HH_Min_Sec.****. Requirement is to extract the date and time from the file name and show the no. of the logs that get generated in an hour.

I read a blog which mentioned that Splunk extracts the timestamp from filename, but that's not happening with this case.

Kindly help with this case. Thanks in advance for any assistance.

Regards, Karthikeyan.SV

Re: Extract data from filename

richgalloway — Mon, 23 Aug 2021 12:46:06 GMT

See https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkextractstimestamps#How_Splunk_software_assigns_timestamps for how Splunk assigns timestamps. This happens at index time. If it isn't happening then make sure the TIME_FORMAT setting is correct (%m_%d_%Y_%H_%M_%S).

You may be able to extract the times at search time.

index=foo source="trace_*" | rex field=source "trace_(?<sourceTime>[^\.]+)" | eval _time=strptime(sourceTime, "%m_%d_%Y_%H_%M_%S") | timechart span=1h count

Re: Extract data from filename

Karthikeyan — Mon, 23 Aug 2021 10:12:29 GMT

Hi,

The proposed extract string worked. The file naming convention is "trace_mm_dd_yyyy_hh_MM_ss_<ramdom data>. txt, where random data like 1342_30bfb72c_9f70_4e32_ad3a_3066dd187240. The random data shown here is an example. The extract string we used extracts the data from filename after trace_ till the end before ".". Is it possible to extract the data between trace_ till _<random data>.txt ?

Karthikeyan

Re: Extract data from filename

richgalloway — Mon, 23 Aug 2021 12:49:40 GMT

Try this regex

trace_(?<sourceTime>\d\d_\d\d_\d\d\d\d_\d\d_\d\d_\d\d)

Re: Extract data from filename

Karthikeyan — Tue, 24 Aug 2021 17:15:49 GMT

I ran into an issue with the search command.

index=foo source="trace_*"
| rex field=source "trace_(?<sourceTime>[^\.]+)"
| eval _time=strptime(sourceTime, "%m_%d_%Y_%H_%M_%S")
| timechart span=1h count

When I use the above command, to see the data for 24 hrs, the count for each hour is displayed which is not a correct value. But when I tried to see events for particular hour, directly by accessing the events of an hour, the same search query was used without timechart and this provides increased no. of events which is actually the correct value to be displayed.

Re: Extract data from filename

richgalloway — Tue, 24 Aug 2021 17:23:44 GMT

Please post a new question for this problem.