How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

nilotpaldutta — Fri, 26 Jun 2015 08:52:36 GMT

I have different environments. In each environment logs are located in different path. e.g.:
C:\Program Files\Splunk...\etc\apps......\abc-20150626123001.log
/u01/splunk/...../etc/apps/...../def-20150626044921.log
/u01/log02/splunk/...../etc/apps/...../mno-20150626071656.log
/u02/splunk/...../etc/apps/...../xyz-20150626044921.log

I am using this query -

index="myindex" | dedup source | sort -source | dedup sourcetype | table sourcetype, source

It gives me the result but contains the complete log path. My requirement is to remove everything and display only the date and time e.g. 2015-06-26 12:30:01

Thanks for answering my previous question [@richgalloway] --> /answers.splunk.com/answers/243218/how-to-use-rex-and-sed-to-insert-and-in-the-result.html
I was able to capture the date and time by using sed and regular expression. e.g.:

index="myindex" | dedup source | sort -source | dedup sourcetype | rex field=source mode=sed "s/[^0-9]*//g" | rex field=source mode=sed "s/(.{4})(.{2})(.{2})/\1-\2-\3 /" | rex field=source mode=sed "s/(.{10})(.{3})(.{2})/\1 \2:\3:/" | table sourcetype, source

This worked only for log files present in "C:\Program Files...."

However it didn't work for other environments. I had to modify each time, e.g. when it's /u01, i used -

index="myindex" | dedup source | sort -source | dedup sourcetype | rex field=source mode=sed "s/01//" | rex field=source mode=sed "s/[^0-9]*//g" | rex field=source mode=sed "s/(.{4})(.{2})(.{2})/\1-\2-\3 /" | rex field=source mode=sed "s/(.{10})(.{3})(.{2})/\1 \2:\3:/" | table sourcetype, source

Basically I just need to capture the YYYYMMDDHHMMSS before .log and remove everything else. The app will be used by various teams. Each team will have their own folder structure. Is there a way to make the query work in any environment irrespective of the path?

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

martin_mueller — Fri, 26 Jun 2015 09:17:27 GMT

How 'bout this?

... | rex field=source "(?<date>\d+)\.log$"

That will grab the last bunch of digits before the .log at the end.

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

nilotpaldutta — Fri, 26 Jun 2015 10:21:08 GMT

Can you please write the whole query?

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

martin_mueller — Fri, 26 Jun 2015 11:02:01 GMT

I don't know what you're trying to achieve with your whole query, so this is just a guess:

| tstats count where index=myindex by source sourcetype | rex field=source "(?<date>\d+)\.log$

topic Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?" in Splunk Search

How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"

Re: How to make the search dynamic - continuation of "How to use rex and sed to insert '-' and ':' in the result?"