Getting Data In

How to edit props.conf and transforms.conf to keep specific events and discard the rest?

Contributor

props.conf

[host::192.168.1.20:514]
TRANSFORMS-set= setnull,sra

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = m=236
DEST_KEY = queue
FORMAT = indexQueue

Basically, I want my Splunk server to accept any event where field "m" equals 236 for a certain host. All other events get discarded. After putting both in "Splunk\etc\system\local" and restarting Splunk, I'm still getting all events.

Here's an example of _raw:

Jan 20 14:49:37 10.11.83.1 id=firewall sn=123456789ABC time="2015-04-17 19:48:01 UTC" fw=192.168.1.96 pri=6 c=16 m=236 msg="An error has occurred" sess="Web" n=11 usr="admin" src=192.168.1.200 dst=192.168.1.95 proto=tcp/800

Esteemed Legend

Try this.

props.conf:

[host::192\.168\.1\.20]
TRANSFORMS-set= setnull,sra

transforms.conf"

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = \s+m=236\s+msg=
DEST_KEY = queue
FORMAT = indexQueue

Put these files on every Indexer and restart all splunkd instances there. Events that arrive and are indexed after this will be correct; older events will stay as they were.

0 Karma

Legend

Hi tmontney,
if in your logs you have "m=236", you can filter your logs in the way you used, but correcting the regex:

 REGEX = m\=236

Beware that if you insert props.conf and transforms.conf in $SPLUNK_HOME/etc/system/local, you cannot manage them by Deployment Server and to deploy them in many servers will be very heavy!
Every way it's a best practice to manage configurations in Apps or Technology Add-Ons.

Bye.
Giuseppe

0 Karma

SplunkTrust
SplunkTrust

The REGEX attribute, during index-time operations, only works fields specified here, and by default applied to _raw. It doesn't apply to any custom field that you might have. If the literal string m=235 is not there in raw data, it won't be effective. You need to write a REGEX which will work on your raw data to filter/drop events that you want. If you can provide some sample events, of both you want to keep and drop, and highlight where the field m value is in there, we may suggest something.

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Transformsconf#KEYS:

0 Karma

Contributor

Also, updated to show an example of an event.

0 Karma

Contributor

You're saying because fields haven't been extracted at this point, yeah? The literal string, m=235, is in there.

0 Karma

SplunkTrust
SplunkTrust

The REGEX should match exact format where that literal string is available, check if there are any extra spaces in between. Also, this should be setup on Indexers/Heavy forwarder.

0 Karma

Contributor

Never mind, I realized I specified the port number in the address.

0 Karma

SplunkTrust
SplunkTrust

So does it work after your remove the port?

0 Karma

Contributor

The thing is, I'd expect to be getting no information. But I'm still getting all. Is my props file correct?

0 Karma

SplunkTrust
SplunkTrust

Is that the exact host name? If all the data that you're receiving is for single sourcetype, use that (is much simpler). Also, How are you getting the data, through Universal forwarder? If yes, then is UF sending data to Indexers or some heavy forwarder? The props/transforms should be placed there. Other than that (location of conf files and validity of stanza name in props.conf), your configurations looks good.

0 Karma

Contributor

I'm getting data from a network device via syslog. I cannot configure what it sends me. It's all or nothing.

0 Karma

SplunkTrust
SplunkTrust

How about you try this for REGEX in transforms.conf

REGEX = \s+m=236\s+
0 Karma

Contributor

Does match exact format. It's one character field name equals a number.

0 Karma