topic Re: Index selection conditional on values in the data in Getting Data In

Index selection conditional on values in the data

davidatpinger — Tue, 29 Sep 2020 07:19:19 GMT

I've got a bunch of key-value data, something sorta like this:

a=1,b=2,c=3,d=4
a=5,b=6,c=7,d=8
a=9,b=2,c=10,d=11
(etc.)

I'd like to sort this data into different indexes (for the purpose of different retention times) depending on the value passed to the 'b' key. So, if b=2, send the data to index_retain_for_one_week but if b=6, send the data to index_retain_for_one_month. Ideally, there would be a final condition for values of 'b' that aren't listed. Think of this as an 'else' condition that sends non-matching data for the list of conditions to index_retain_one_day. (All of the index names are just illustrative, like the data.)

Is there a good way to do that? Heck, is it possible? My apologies if this is already answered somewhere - I couldn't find a set of key words that generated an answer. Thanks!

Re: Index selection conditional on values in the data

somesoni2 — Mon, 21 Sep 2015 21:25:41 GMT

Here you go

http://answers.splunk.com/answers/150266/dynamic-index-assignment-based-on-event-or-log-prefix.html

Re: Index selection conditional on values in the data

davidatpinger — Mon, 21 Sep 2015 22:13:12 GMT

I suppose I can make multiple stanzas of transforms and they are applied in order by the TRANSFORMS statement in props.conf. So yeah, this will work! Thanks!

Re: Index selection conditional on values in the data

davidatpinger — Tue, 29 Sep 2020 07:19:27 GMT

Okay, this is close but not quite working. I've got something like this in transforms.conf:

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
DEST_KEY=_MetaData:Index
FORMAT = normal-index

And then, in props.conf, I have something like this:

[mysourcetype]
TRANSFORMS-indexsort = special-data, normal-data

There must be something unhappy with the REGEX, because everything ends up in special-index. Hmm.

Re: Index selection conditional on values in the data

somesoni2 — Tue, 22 Sep 2015 18:36:42 GMT

You forgot to add "REGEX = ." in the normal-data stanza.

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
REGEX = .
DEST_KEY=_MetaData:Index
FORMAT = normal-index

Re: Index selection conditional on values in the data

davidatpinger — Tue, 22 Sep 2015 18:46:42 GMT

Yeah, and I need parens instead of square brackets. Getting there! (Thanks!!)

Re: Index selection conditional on values in the data

davidatpinger — Tue, 22 Sep 2015 19:52:01 GMT

Hmm, now everything is falling through to normal-data. Time to muck around with it some more.

Re: Index selection conditional on values in the data

somesoni2 — Tue, 22 Sep 2015 20:18:10 GMT

Check the REGEX for special-data, may be some spaces that need to be adjusted etc. If you can send some actual sample data, I can try to look at it as well..

Re: Index selection conditional on values in the data

somesoni2 — Tue, 22 Sep 2015 20:22:40 GMT

Also, can you do this, in your porps.conf and transforms.conf, change the order of the stanza, so get the normal-data first and special-data after that.

Re: Index selection conditional on values in the data

davidatpinger — Fri, 02 Oct 2015 21:40:22 GMT

I discovered that my brain was backwards. It's not first-match and stop in the listed transforms in props.conf - it runs to the end and the last match is what you get.

Once I got that through my skull, everything works as expected. Thanks much!