topic Regular expression in source in Splunk Search

Regular expression in source

gimbil — Mon, 11 Nov 2013 19:10:10 GMT

Hi All,

I have multiple sources like
a1.gz
a2.gz
a3.gz
a4.gz
a5.gz
…

and so one. How can I have a subset these as source in search? I want to have something like
source=a[1-3]* | stats …

to use a1.gz, a2.gz and a3.gz as source but it does not accept [1-3] and gives error.

Thanks

Re: Regular expression in source

Ayn — Mon, 11 Nov 2013 19:27:50 GMT

The search command does not support filtering using regexes. You'll either have to filter using wildcards and/or explicit individual terms, or use the separate regex operator as your second command, like this:

source=a* | regex source="a[1-3]*"

The drawback to this approach is that Splunk will read all events matching source=a* first before sending them to the regex command that then performs the filtering, which means more events will be read from disk than what is strictly necessary.

Re: Regular expression in source

richgalloway — Mon, 11 Nov 2013 19:32:35 GMT

You can use wildcards in the source specifier, but (AFAIK) not regex. So 'source="a*.gz" | ...' should work.

There's also '| where like(source,"a%.gz") | ...', but that's still not regex and is probably less efficient.

Finally, you can try '| regex source="a[0-9]+.gz" | stats ...', but that's probably still less efficient that the first option.

Re: Regular expression in source

gimbil — Mon, 11 Nov 2013 19:36:48 GMT

Sorry but for some reason source=a* | regex source="b" is still returning everything and not filtering at all. Am I missing something?

Re: Regular expression in source

Ayn — Mon, 11 Nov 2013 20:04:28 GMT

No that looks about right, depending on how your filter looks more exactly and what sources you expect to be filtered out.

Re: Regular expression in source

landen99 — Thu, 10 Sep 2015 19:44:09 GMT

Ayn did not say "b". I would refine her answer to:

source=a* | regex source="a[1-3].gz"

Re: Regular expression in source

somesoni2 — Thu, 10 Sep 2015 21:06:56 GMT

Best option is to use the wildcard to get you filter done, as specified by @Ayn and @richgalloway. The regex options may be inefficient based on your data distribution among the source and filter, however, another option that you can try is to specify the required source name in the base search, using subsearch, something like this

index=blah [| metadata type=sources index=blah | table source | regex source="a[1-3].gz" ] | rest of the search

The subsearch will grab all the required source (a1.gz/a2.gz/a3.gz in your example) and generate an OR condition into the base search, so effectively your search will become:-

 index=blah ((source=a1.gz) OR (source=a2.gz) OR (source=a3.gz)) | rest of the search