topic Re: Getting date from filename and time from logs in Getting Data In

Getting date from filename and time from logs

silvermail — Fri, 26 Nov 2010 14:46:45 GMT

Hello,

I am not sure if this is possible, but I have a file named called php_201000618.txt and inside the logs it contain something like:

----------Time 20:54:17----------
HTTP/1.1 200 OK
Date: Sun, 6 Jun 2010 19:33:17 GMT
Server: Apache
Content-Length: 51
Keep-Alive: timeout=300, max=636

I need to get the date from the filename 20100618 and the time from 20:54:17. Is it possible to split the timestamp recognition precedence into two components?

I know it may seem easier to get it from the line "Date: Sun, 6 Jun 2010 19:33:17 GMT", but this line does not exist in all the lines - and the only reliable place is to get the date from the filename as well as the header "----------Time 20:54:17----------"

Right now I have a props.conf that shows:

TIME_PREFIX = ^----------Time
TIME_FORMAT = %H:%M:%S

Time extraction is working fine, but I am getting some random dates which I am not sure why this is happening.

Thanks for any suggestions.

Re: Getting date from filename and time from logs

Rob — Fri, 26 Nov 2010 21:09:18 GMT

You may want to take a look at the following answer that has been posted. I believe that it answers your question as well.

http://splunk-base.splunk.com/answers/12015/setting-date-on-event-based-on-filename

Re: Getting date from filename and time from logs

silvermail — Fri, 26 Nov 2010 23:13:22 GMT

Thanks Rob, but the example you have mentioned isn't what I was looking for. Thanks anyway...

Re: Getting date from filename and time from logs

gkanapathy — Tue, 30 Nov 2010 04:45:35 GMT

Try this: http://blogs.splunk.com/2009/12/02/configure-splunk-to-pull-a-date-out-of-a-non-standard-filename/

Yours might not be so complicated, but the general idea holds.

Re: Getting date from filename and time from logs

silvermail — Tue, 30 Nov 2010 22:08:31 GMT

Thanks Gerald. I have my own datetime.xml defined already, and it's working great for some events, but for certain events Splunk is pulling the date from some other values in the logs.

So that is the reason why I'm asking if it's possible to change the timestamp recognition precedence so that it will hit my datetime.xml first before going to some other places to look for the date.

Thanks for the response though, always appreciative of your help.

Re: Getting date from filename and time from logs

gkanapathy — Wed, 01 Dec 2010 10:37:10 GMT

There is not, but if you make sure the datetime.xml contains only date formats that will definitely fail in the file, then it should fall through to the filename.

Re: Getting date from filename and time from logs

hexx — Mon, 23 May 2011 13:43:16 GMT

The link provided by Rob has been "updated" to better suit your use-case 🙂 Take a look at it now!

Re: Getting date from filename and time from logs

kenkenou — Tue, 29 Sep 2020 11:35:31 GMT

I set TIME_PREFIX TIME_FORMAT and plus LOOK_AHEAD(how long the time string is) then ,it works well.