Getting Data In

Index selection conditional on values in the data

Path Finder

I've got a bunch of key-value data, something sorta like this:

a=1,b=2,c=3,d=4
a=5,b=6,c=7,d=8
a=9,b=2,c=10,d=11
(etc.)

I'd like to sort this data into different indexes (for the purpose of different retention times) depending on the value passed to the 'b' key. So, if b=2, send the data to index_retain_for_one_week but if b=6, send the data to index_retain_for_one_month. Ideally, there would be a final condition for values of 'b' that aren't listed. Think of this as an 'else' condition that sends non-matching data for the list of conditions to index_retain_one_day. (All of the index names are just illustrative, like the data.)

Is there a good way to do that? Heck, is it possible? My apologies if this is already answered somewhere - I couldn't find a set of key words that generated an answer. Thanks!

0 Karma
1 Solution

Path Finder

I suppose I can make multiple stanzas of transforms and they are applied in order by the TRANSFORMS statement in props.conf. So yeah, this will work! Thanks!

0 Karma

Path Finder

Okay, this is close but not quite working. I've got something like this in transforms.conf:

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
DEST_KEY=_MetaData:Index
FORMAT = normal-index

And then, in props.conf, I have something like this:

[mysourcetype]
TRANSFORMS-indexsort = special-data, normal-data

There must be something unhappy with the REGEX, because everything ends up in special-index. Hmm.

0 Karma

Revered Legend

You forgot to add "REGEX = ." in the normal-data stanza.

[special-data]
DEST_KEY = _MetaData:Index
REGEX = b=[3|30|44|49|21]
FORMAT = special-index

[normal-data]
REGEX = .
DEST_KEY=_MetaData:Index
FORMAT = normal-index
0 Karma

Path Finder

Yeah, and I need parens instead of square brackets. Getting there! (Thanks!!)

0 Karma

Path Finder

Hmm, now everything is falling through to normal-data. Time to muck around with it some more.

0 Karma

Revered Legend

Check the REGEX for special-data, may be some spaces that need to be adjusted etc. If you can send some actual sample data, I can try to look at it as well..

0 Karma

Revered Legend

Also, can you do this, in your porps.conf and transforms.conf, change the order of the stanza, so get the normal-data first and special-data after that.

0 Karma

Path Finder

I discovered that my brain was backwards. It's not first-match and stop in the listed transforms in props.conf - it runs to the end and the last match is what you get.

Once I got that through my skull, everything works as expected. Thanks much!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!