topic How to search multiple sources within my search? in Splunk Search

How to search multiple sources within my search?

andreacorrie — Tue, 13 Sep 2016 22:12:15 GMT

How do I search multiple source files within my search? I want to do something like:

source="/foo/bar/2016/09/{08,15}/*.avro"

source="/foo/bar/2016/09/[08-30]/*.avro"

but neither syntax returns results.

Re: How to search multiple sources within my search?

skoelpin — Tue, 13 Sep 2016 22:15:44 GMT

You could do source=* or you could do source="/foo/bar/2016/09/{08,09}/*.avro" OR source="/foo/bar/2016/09/[08-09]/*.avro"

Re: How to search multiple sources within my search?

andreacorrie — Tue, 13 Sep 2016 22:19:10 GMT

In my example, I want to search the 8th or the 9th. I'm using syntax you would use in Apache Pig to specify multiple days. Yes, I could use source = source="/foo/bar/2016/09/08/*.avro OR source="/foo/bar/2016/09/09/*.avro but this doesn't scale if you want to search more than a couple days.

Re: How to search multiple sources within my search?

skoelpin — Tue, 13 Sep 2016 22:28:44 GMT

Ahh I see.. You want to have the ability to scale your searches relative to the date which is a dynamic value. Are the days in your source offset from the current day or do they match up to the current date?

I'm assuming with this, you want to search the data from that source that is from Sept 8 and Spet 9th?

source="/foo/bar/2016/09/{08,09}/*.avro"

Re: How to search multiple sources within my search?

kschon_splunk — Tue, 13 Sep 2016 22:57:40 GMT

In the example you are using, I would suggest extracting the _time variable from your path, and then restricting your query by time (e.g. using the graphical time range picker). There is more information here:
https://docs.splunk.com/Documentation/Hunk/6.4.3/Hunk/Virtualindexconfigurationvariables

In particular, look at these properties:

vix.input.[N].et.regex
vix.input.[N].et.format
vix.input.[N].lt.regex
vix.input.[N].lt.format

If you want to query on something other than time, you can extract additional variables from the path as well, using the property "vix.input.1.path". You can see some examples here:
https://docs.splunk.com/Documentation/Hunk/6.4.3/Hunk/Setupavirtualindex

That should simplify your queries somewhat, since you won't have to simultaneously specify the values you're looking for, and where to find them in the path.

Re: How to search multiple sources within my search?

pasokkum — Mon, 19 Sep 2016 06:30:14 GMT

try this..

Re: How to search multiple sources within my search?

andreacorrie — Mon, 19 Sep 2016 20:26:20 GMT

This will work but again, it doesn't scale for more than a few days. I'm looking for a solution that I can search 30 days, for example.

Re: How to search multiple sources within my search?

sundareshr — Mon, 19 Sep 2016 20:58:48 GMT

Try this

base search | regex source="\/\d{2}\/(0[8-9]|1[0-5])\/"

Re: How to search multiple sources within my search?

somesoni2 — Mon, 19 Sep 2016 21:08:48 GMT

This should work

index=foo sourcetype=bar [| gentimes start=-1 | eval t=mvrange(8,30+1) | table t | mvexpand t | eval source="/foo/bar/2016/09/".substr("0".t,-2) | table source] | .... rest of the search

The subsearch is generating a series of sources from "/foo/bar/2016/09/08" to "/foo/bar/2016/09/30" in a giant OR statement ( source="/foo/bar/2016/09/08" OR source="/foo/bar/2016/09/09"...OR "/foo/bar/2016/09/30")