Splunk Search

How to configure props.conf and transforms.conf to route data matching a certain regex to a specific index and drop all other events?

Sloefke
Path Finder

Hi,

I'm running a test setup with some live syslog data and I want to do the following on my forwarder:

1) Route all data matching a certain regex to a specific index on my indexer
2) Drop all other events

I already have been playing around with the _MetaData:Index key which seems to work just fine when applied as single transform for a certain source. However, combining it with a general drop transform seems to be different.

This is my props.conf, with the transforms in the order the documentation is telling me to put them (http://docs.splunk.com/Documentation/Splunk/6.1.3/Forwarding/Routeandfilterdatad#Keep_specific_event...

[source::/var/log/splunk*]
TRANSFORMS-syslog = drop,test

When testing the same syntax as mentioned in the doc, the filtering works and all events matching the "test" transform get sent to the main index:

[drop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[test]
REGEX = <regex>
DEST_KEY = queue
FORMAT = indexQueue

However, when I want to route to the test index, the "test" transform just doesn't seem to get applied. All data is dropped since my indexes remain empty. transforms.conf in that case:

[drop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[test]
REGEX = <regex>
DEST_KEY = _MetaData:Index
FORMAT = test

Is combining these two transforms possible? If yes, how?
Or do I need to route the filtered data to the indexQueue and then route it again to the correct index somewhere else?

1 Solution

martin_mueller
SplunkTrust
SplunkTrust

In the second pair of transforms rules, you're telling Splunk to drop everything into the nullQueue and to change the index on some events. That index change becomes irrelevant though because everything is sent to the nullQueue, the test stanza doesn't change the queue field.

You should succeed with the drop rule and both test rules. First, drop everything. Second, un-drop events that match your regex (test stanza from the first bit). Third, set the index however you need (test stanza from second bit).

View solution in original post

becarimi
New Member

I'm trying to do the same but I do not get the expected result.

I receive syslog messages by upd: 514, I want to do is to filter syslog messages with a particular word and discard the rest messages.

The settings of my props.conf is this:

[syslog]
pulldown_type = true
maxdist = 3
TIME_FORMAT =% b% d% H:% M:% S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
TRANSFORMS-set = otroindex,todos
#TRANSFORMS-null = otroindex,todos

Transform.conf the configuration of this file is:

[otroindex]
REGEX = miguel
DEST_KEY = _MetaData: Index
FORMAT = test
WRITE_META = true

[todos]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

But not quite work, if in the fields:

DEST_KEY = queue
FORMAT = nullQueue

I do not receive any data.

But if I write the following works properly:

REGEX = <keyword that you want>
DEST_KEY = _MetaData: Index
FORMAT = <index you want>
WRITE_META = true

How can I do to filter syslog messages I want and discard the other messages(dont save) ??

,I'm trying to do the same but I do not get the expected result.

I receive syslog messages by upd: 514, I want to do is to filter syslog messages with a particular word and discard the rest messages.

The settings of my props.conf is this:

[syslog]
pulldown_type = true
maxdist = 3
TIME_FORMAT =% b% d% H:% M:% S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
TRANSFORMS-set = otroindex,todos
#TRANSFORMS-null = otroindex,todos

Transform.conf the configuration of this file is:

[otroindex]
REGEX = miguel
DEST_KEY = _MetaData: Index
FORMAT = test
WRITE_META = true

[todos]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

But not quite work, if in the fields:

DEST_KEY = queue
FORMAT = nullQueue

I do not receive any data.

But if I write the following works properly:

REGEX = <keyword that you want>
DEST_KEY = _MetaData: Index
FORMAT = <index you want>
WRITE_META = true

How can I do to filter syslog messages I want and discard the other messages(dont save) ??

0 Karma

piebob
Motivator

please consider asking this as a separate question.

martin_mueller
SplunkTrust
SplunkTrust

In the second pair of transforms rules, you're telling Splunk to drop everything into the nullQueue and to change the index on some events. That index change becomes irrelevant though because everything is sent to the nullQueue, the test stanza doesn't change the queue field.

You should succeed with the drop rule and both test rules. First, drop everything. Second, un-drop events that match your regex (test stanza from the first bit). Third, set the index however you need (test stanza from second bit).

Sloefke
Path Finder

Seems I didn't get the transforms logic completely, your suggestion works perfectly. Thanks!

0 Karma

tyron_
Explorer

Expanding on @martin_mueller 's answer, it should be similar to:

 [drop]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

 [undrop-test]
 REGEX = <regex>
 DEST_KEY = queue
 FORMAT = indexQueue

 [test]
 REGEX = <regex>
 DEST_KEY = _MetaData:Index
 FORMAT = test
0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...