Getting Data In

Filter out logs using props.conf and transfors.conf

daniel_augustyn
Contributor

I am pulling logs from the firewalls via scripts on a heavy forwarder (via scrips from the app for Checkpoint). How to create props.conf and transfoms.conf to filter some logs from being indexed by the indexers. And where to put them? In the $Splunk/etc/apps/APP_NAME/local folder or in the $SPLUNK/etc/system/local/ folder on the heavy forwarder?

This is what I've got so far and it doesn't seem to be picking up the logs that I want to filter out.

props.conf:
[source::...opsec]
sourcetype = opsec

[opsec]
TRANSFORMS-set= setnull, setparsing

transforms.conf
[setnull]
REGEX = LAB
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

0 Karma
1 Solution

masonmorales
Influencer

I believe sourcetype renaming is only applied at search time, so the [opsec] stanza in props.conf would not be picked up during the parsing phase.

What happens if you change your props.conf to:
[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

If you still have problems, try adding the config to your indexers too. To rename the sourcetype, add a props.conf to your search head(s):
[source::...opsec]
sourcetype = opsec

View solution in original post

masonmorales
Influencer

I believe sourcetype renaming is only applied at search time, so the [opsec] stanza in props.conf would not be picked up during the parsing phase.

What happens if you change your props.conf to:
[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

If you still have problems, try adding the config to your indexers too. To rename the sourcetype, add a props.conf to your search head(s):
[source::...opsec]
sourcetype = opsec

daniel_augustyn
Contributor

It started picking up after I had deleted these two files and created new ones. And after I rebooted the heavy forwarder. I still don't know what was the issue at the first place, since the files look identical and I was rebooting Splunk before after each change.

0 Karma

daniel_augustyn
Contributor

It still doesn't seem to be picking up the events with "LAB" word in them. Is there something wrong with the code in any of these files?

0 Karma

Jeremiah
Motivator

You can put your props.conf and transforms.conf in an app or under system/local. The system/local directory will win out over anything you have set it an app.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles

Using an app is generally a good idea, because it allows you to package and re-deploy it if you need to. Some people will put all of their index-time props and transforms in a single app, others break them up by technology or application. Thats more a matter of preference and what works best for you.

For the settings you have here, I would do one of two things. Either set the sourcetype in your inputs.conf file so you don't have to set it in your props.conf, or move your TRANSFORM to your source stanza:

[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

daniel_augustyn
Contributor

The regex doesn't seem to be picking up the events I want to filter out: REGEX = LAB

I am still getting all of the event with "LAB" word indexed.

0 Karma

Jeremiah
Motivator

Actually looking at what you have, since you want to drop events with "LAB", you just need the setnull transform, not the setparsing.

TRANSFORMS-set = setnull

You can see something similar here:

https://answers.splunk.com/answers/107605/filtering-events-out-via-props-conf-and-transforms-conf.ht...
and here
https://answers.splunk.com/answers/293599/how-to-configure-propsconf-and-transformsconf-to-f-2.html

0 Karma

daniel_augustyn
Contributor

Still doesn't pick up the events I want to filter out. Is this something off with this:
[source::...opsec]
sourcetype = opsec

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...