topic Re: Extract date from Filename source in Getting Data In

Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 07:15:16 GMT

Hi in my events I am getting time which is extracted correctly by Splunk for _time timestamp and for date extraction I want to use source field which is in format like- D:\\abc\\def\\XYLog09229190601.txt.zip:.\\XYLog09229190601.txt
From above source field I want to extract last 6 digit before .txt to create date like for this example it will be 2019-06-01.
What will be best way to do that..
Is it using datetime.xml? and how I can achieve it since for year only two digits are mentioned.
Thanks.

Re: Extract date from Filename source

DavidHourani — Mon, 10 Jun 2019 08:07:55 GMT

Hi @ips_mandar,

Which version are you running ?

If Splunk 7.2+, have a look here :
https://docs.splunk.com/Documentation/Splunk/latest/Data/IngestEval

This is a pretty easy way to get time extracted at index time.

Cheers,
David

Re: Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 08:39:33 GMT

Thanks For your suggestion @DavidHourani ..Yes I am using Splunk 7.2+
But I thought that it will impact performance so I am trying to avoid it (Ingest Eval).

Re: Extract date from Filename source

DavidHourani — Mon, 10 Jun 2019 08:47:08 GMT

Yeah true, there would be an impact... In that case have you tried simply setting TIME_FORMAT in your props.conf ?
This will first apply to the events and then fallback to the filename if it's not found in the event, so it should grab your time from there.

Re: Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 09:22:35 GMT

Currently I am trying with changing datetime.xml and if that fails then I will try TIME_FORMAT as well. So is there any performance impact if I use separate datetime.xml for app?

Re: Extract date from Filename source

DavidHourani — Mon, 10 Jun 2019 09:26:11 GMT

Yeap you can also do it with datetime.xml it's just as good as TIME_FORMAT, I thought you were looking for an easy solution that's why I didn't mention datetime.xml, if you have that working stick to it ^^

Re: Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 09:55:12 GMT

HI @DavidHourani ,
I want one day before of extracted date as _time How I can achieve this?
like from source I have extracted as 10 June then I want _time as 09 June..

Re: Extract date from Filename source

DavidHourani — Mon, 10 Jun 2019 10:12:46 GMT

You can do this at search time with something like this : ...| eval _time=_time-86400
Or save that as a calculated field to have it permanently applied at search time.
Best would be to go back to IngestEval it's the only way to apply an eval at index time..
Or if you can manipulate your filenames, then fix it to day -1 🙂

Re: Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 13:14:39 GMT

Thanks @DavidHourani
My current problem:-
my event looks like-

31,04:56:47:928,  abc:0xabc,  49.716720, -59.271553,197

where 31 is day of month
and source field looks like - D:\\abc\\def\\XYLog09229190601.txt.zip:.\\XYLog09229190601.txt
So by which way I can get _time as 31 May 2019?

Re: Extract date from Filename source

ips_mandar — Mon, 10 Jun 2019 13:20:51 GMT

and for same source some events looks like

30,04:56:47:928,  abc:0xabc,  49.716720, -59.271553,197

For which I want _time as 30 May 2019

Re: Extract date from Filename source

DavidHourani — Mon, 10 Jun 2019 13:53:50 GMT

ouch, so same source, different ways to extract time, is that what you mean ? coz for both events you posted it seems you can get everything from the event itself, except the month, right ?

Re: Extract date from Filename source

DavidHourani — Tue, 11 Jun 2019 06:14:46 GMT

refer here for the exact eval to be used :
https://answers.splunk.com/answers/751102/extract-date-timestamp-from-raw-data-and-source-fi.html