Regular expression in source

[gimbil]

Hi All,

I have multiple sources like
a1.gz
a2.gz
a3.gz
a4.gz
a5.gz
…

and so one. How can I have a subset these as source in search? I want to have something like
source=a[1-3]* | stats …

to use a1.gz, a2.gz and a3.gz as source but it does not accept [1-3] and gives error.

Thanks

[somesoni2]

Best option is to use the wildcard to get you filter done, as specified by @Ayn and @richgalloway. The regex options may be inefficient based on your data distribution among the source and filter, however, another option that you can try is to specify the required source name in the base search, using subsearch, something like this

index=blah [| metadata type=sources index=blah | table source | regex source="a[1-3].gz" ] | rest of the search

The subsearch will grab all the required source (a1.gz/a2.gz/a3.gz in your example) and generate an OR condition into the base search, so effectively your search will become:-

 index=blah ((source=a1.gz) OR (source=a2.gz) OR (source=a3.gz)) | rest of the search

[richgalloway]

You can use wildcards in the source specifier, but (AFAIK) not regex. So 'source="a*.gz" | ...' should work.

There's also '| where like(source,"a%.gz") | ...', but that's still not regex and is probably less efficient.

Finally, you can try '| regex source="a[0-9]+.gz" | stats ...', but that's probably still less efficient that the first option.

[Ayn]

The search command does not support filtering using regexes. You'll either have to filter using wildcards and/or explicit individual terms, or use the separate regex operator as your second command, like this:

source=a* | regex source="a[1-3]*"

The drawback to this approach is that Splunk will read all events matching source=a* first before sending them to the regex command that then performs the filtering, which means more events will be read from disk than what is strictly necessary.

[Ayn]

No that looks about right, depending on how your filter looks more exactly and what sources you expect to be filtered out.

[gimbil]

Sorry but for some reason source=a* | regex source="b" is still returning everything and not filtering at all. Am I missing something?

[landen99]

Ayn did not say "b". I would refine her answer to:

source=a* | regex source="a[1-3].gz"