topic Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest? in Getting Data In

How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 21:45:24 GMT

props.conf

[host::192.168.1.20:514]
TRANSFORMS-set= setnull,sra

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = m=236
DEST_KEY = queue
FORMAT = indexQueue

Basically, I want my Splunk server to accept any event where field "m" equals 236 for a certain host. All other events get discarded. After putting both in "Splunk\etc\system\local" and restarting Splunk, I'm still getting all events.

Here's an example of _raw:

Jan 20 14:49:37 10.11.83.1 id=firewall sn=123456789ABC time="2015-04-17 19:48:01 UTC" fw=192.168.1.96 pri=6 c=16 m=236 msg="An error has occurred" sess="Web" n=11 usr="admin" src=192.168.1.200 dst=192.168.1.95 proto=tcp/800

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

somesoni2 — Thu, 26 Jan 2017 21:52:42 GMT

The REGEX attribute, during index-time operations, only works fields specified here, and by default applied to _raw. It doesn't apply to any custom field that you might have. If the literal string m=235 is not there in raw data, it won't be effective. You need to write a REGEX which will work on your raw data to filter/drop events that you want. If you can provide some sample events, of both you want to keep and drop, and highlight where the field m value is in there, we may suggest something.

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Transformsconf#KEYS:

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 21:55:31 GMT

You're saying because fields haven't been extracted at this point, yeah? The literal string, m=235, is in there.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 21:59:05 GMT

Also, updated to show an example of an event.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

somesoni2 — Thu, 26 Jan 2017 22:01:01 GMT

The REGEX should match exact format where that literal string is available, check if there are any extra spaces in between. Also, this should be setup on Indexers/Heavy forwarder.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 22:04:55 GMT

Does match exact format. It's one character field name equals a number.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 22:07:34 GMT

The thing is, I'd expect to be getting no information. But I'm still getting all. Is my props file correct?

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

somesoni2 — Thu, 26 Jan 2017 22:10:43 GMT

Is that the exact host name? If all the data that you're receiving is for single sourcetype, use that (is much simpler). Also, How are you getting the data, through Universal forwarder? If yes, then is UF sending data to Indexers or some heavy forwarder? The props/transforms should be placed there. Other than that (location of conf files and validity of stanza name in props.conf), your configurations looks good.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

somesoni2 — Thu, 26 Jan 2017 22:12:28 GMT

How about you try this for REGEX in transforms.conf

REGEX = \s+m=236\s+

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 22:12:39 GMT

Never mind, I realized I specified the port number in the address.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

tmontney — Thu, 26 Jan 2017 22:15:58 GMT

I'm getting data from a network device via syslog. I cannot configure what it sends me. It's all or nothing.

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

somesoni2 — Thu, 26 Jan 2017 22:25:15 GMT

So does it work after your remove the port?

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

gcusello — Fri, 27 Jan 2017 07:34:36 GMT

Hi tmontney,
if in your logs you have "m=236", you can filter your logs in the way you used, but correcting the regex:

 REGEX = m\=236

Beware that if you insert props.conf and transforms.conf in $SPLUNK_HOME/etc/system/local, you cannot manage them by Deployment Server and to deploy them in many servers will be very heavy!
Every way it's a best practice to manage configurations in Apps or Technology Add-Ons.

Bye.
Giuseppe

Re: How to edit props.conf and transforms.conf to keep specific events and discard the rest?

woodcock — Sun, 05 Mar 2017 06:54:02 GMT

Try this.

props.conf:

[host::192\.168\.1\.20]
TRANSFORMS-set= setnull,sra

transforms.conf"

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sra]
REGEX = \s+m=236\s+msg=
DEST_KEY = queue
FORMAT = indexQueue

Put these files on every Indexer and restart all splunkd instances there. Events that arrive and are indexed after this will be correct; older events will stay as they were.