topic Re: Filter out logs using props.conf and transfors.conf in Getting Data In

Filter out logs using props.conf and transfors.conf

daniel_augustyn — Sat, 20 Feb 2016 01:35:11 GMT

I am pulling logs from the firewalls via scripts on a heavy forwarder (via scrips from the app for Checkpoint). How to create props.conf and transfoms.conf to filter some logs from being indexed by the indexers. And where to put them? In the $Splunk/etc/apps/APP_NAME/local folder or in the $SPLUNK/etc/system/local/ folder on the heavy forwarder?

This is what I've got so far and it doesn't seem to be picking up the logs that I want to filter out.

props.conf:
[source::...opsec]
sourcetype = opsec

[opsec]
TRANSFORMS-set= setnull, setparsing

transforms.conf
[setnull]
REGEX = LAB
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Re: Filter out logs using props.conf and transfors.conf

Jeremiah — Sat, 20 Feb 2016 16:57:51 GMT

You can put your props.conf and transforms.conf in an app or under system/local. The system/local directory will win out over anything you have set it an app.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles

Using an app is generally a good idea, because it allows you to package and re-deploy it if you need to. Some people will put all of their index-time props and transforms in a single app, others break them up by technology or application. Thats more a matter of preference and what works best for you.

For the settings you have here, I would do one of two things. Either set the sourcetype in your inputs.conf file so you don't have to set it in your props.conf, or move your TRANSFORM to your source stanza:

[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

Re: Filter out logs using props.conf and transfors.conf

daniel_augustyn — Sat, 20 Feb 2016 21:15:54 GMT

The regex doesn't seem to be picking up the events I want to filter out: REGEX = LAB

I am still getting all of the event with "LAB" word indexed.

Re: Filter out logs using props.conf and transfors.conf

daniel_augustyn — Mon, 22 Feb 2016 05:56:55 GMT

It still doesn't seem to be picking up the events with "LAB" word in them. Is there something wrong with the code in any of these files?

Re: Filter out logs using props.conf and transfors.conf

Jeremiah — Mon, 22 Feb 2016 06:25:18 GMT

Actually looking at what you have, since you want to drop events with "LAB", you just need the setnull transform, not the setparsing.

TRANSFORMS-set = setnull

You can see something similar here:

https://answers.splunk.com/answers/107605/filtering-events-out-via-props-conf-and-transforms-conf.html
and here
https://answers.splunk.com/answers/293599/how-to-configure-propsconf-and-transformsconf-to-f-2.html

Re: Filter out logs using props.conf and transfors.conf

daniel_augustyn — Mon, 22 Feb 2016 20:59:57 GMT

Still doesn't pick up the events I want to filter out. Is this something off with this:
[source::...opsec]
sourcetype = opsec

Re: Filter out logs using props.conf and transfors.conf

masonmorales — Mon, 22 Feb 2016 21:41:43 GMT

I believe sourcetype renaming is only applied at search time, so the [opsec] stanza in props.conf would not be picked up during the parsing phase.

What happens if you change your props.conf to:
[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

If you still have problems, try adding the config to your indexers too. To rename the sourcetype, add a props.conf to your search head(s):
[source::...opsec]
sourcetype = opsec

Re: Filter out logs using props.conf and transfors.conf

daniel_augustyn — Mon, 22 Feb 2016 23:01:24 GMT

It started picking up after I had deleted these two files and created new ones. And after I rebooted the heavy forwarder. I still don't know what was the issue at the first place, since the files look identical and I was rebooting Splunk before after each change.