How to configure props.conf and transforms.conf to route data matching a certain regex to a specific index and drop all other events?

[Sloefke]

Hi,

I'm running a test setup with some live syslog data and I want to do the following on my forwarder:

1) Route all data matching a certain regex to a specific index on my indexer
2) Drop all other events

I already have been playing around with the _MetaData:Index key which seems to work just fine when applied as single transform for a certain source. However, combining it with a general drop transform seems to be different.

This is my props.conf, with the transforms in the order the documentation is telling me to put them (http://docs.splunk.com/Documentation/Splunk/6.1.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

[source::/var/log/splunk*]
TRANSFORMS-syslog = drop,test

When testing the same syntax as mentioned in the doc, the filtering works and all events matching the "test" transform get sent to the main index:

[drop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[test]
REGEX = <regex>
DEST_KEY = queue
FORMAT = indexQueue

However, when I want to route to the test index, the "test" transform just doesn't seem to get applied. All data is dropped since my indexes remain empty. transforms.conf in that case:

[drop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[test]
REGEX = <regex>
DEST_KEY = _MetaData:Index
FORMAT = test

Is combining these two transforms possible? If yes, how?
Or do I need to route the filtered data to the indexQueue and then route it again to the correct index somewhere else?

[becarimi]

I'm trying to do the same but I do not get the expected result.

I receive syslog messages by upd: 514, I want to do is to filter syslog messages with a particular word and discard the rest messages.

The settings of my props.conf is this:

[syslog]
pulldown_type = true
maxdist = 3
TIME_FORMAT =% b% d% H:% M:% S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
TRANSFORMS-set = otroindex,todos
#TRANSFORMS-null = otroindex,todos

Transform.conf the configuration of this file is:

[otroindex]
REGEX = miguel
DEST_KEY = _MetaData: Index
FORMAT = test
WRITE_META = true

[todos]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

But not quite work, if in the fields:

DEST_KEY = queue
FORMAT = nullQueue

I do not receive any data.

But if I write the following works properly:

REGEX = <keyword that you want>
DEST_KEY = _MetaData: Index
FORMAT = <index you want>
WRITE_META = true

How can I do to filter syslog messages I want and discard the other messages(dont save) ??

,I'm trying to do the same but I do not get the expected result.

I receive syslog messages by upd: 514, I want to do is to filter syslog messages with a particular word and discard the rest messages.

The settings of my props.conf is this:

[syslog]
pulldown_type = true
maxdist = 3
TIME_FORMAT =% b% d% H:% M:% S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
TRANSFORMS-set = otroindex,todos
#TRANSFORMS-null = otroindex,todos

Transform.conf the configuration of this file is:

[otroindex]
REGEX = miguel
DEST_KEY = _MetaData: Index
FORMAT = test
WRITE_META = true

[todos]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

But not quite work, if in the fields:

DEST_KEY = queue
FORMAT = nullQueue

I do not receive any data.

But if I write the following works properly:

REGEX = <keyword that you want>
DEST_KEY = _MetaData: Index
FORMAT = <index you want>
WRITE_META = true

How can I do to filter syslog messages I want and discard the other messages(dont save) ??

[piebob]

please consider asking this as a separate question.

[martin_mueller · Accepted solution]

In the second pair of transforms rules, you're telling Splunk to drop everything into the nullQueue and to change the index on some events. That index change becomes irrelevant though because everything is sent to the nullQueue, the test stanza doesn't change the queue field.

You should succeed with the drop rule and both test rules. First, drop everything. Second, un-drop events that match your regex (test stanza from the first bit). Third, set the index however you need (test stanza from second bit).

[Sloefke]

Seems I didn't get the transforms logic completely, your suggestion works perfectly. Thanks!

[tyron_]

Expanding on @martin_mueller 's answer, it should be similar to:

 [drop]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 [undrop-test]
 REGEX = <regex>
 DEST_KEY = queue
 FORMAT = indexQueue

 [test]
 REGEX = <regex>
 DEST_KEY = _MetaData:Index
 FORMAT = test