topic Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest in Getting Data In

How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

merrelr — Wed, 22 Feb 2017 15:55:20 GMT

props.conf

[firewall]
TRANSFORMS-set = setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = 192\.168\.1\.1
DEST_KEY = queue
FORMAT = indexQueue

I have a heavy forwarder with the following. What I want to do is only forward events that match the regex to our indexers for indexing and discard the rest. It doesn't matter what put in the REGEX section though nothing comes through even if I look at the logs and see that there are definitely matches.

If I change props.conf to

TRANSFORMS-set = setparsing

I get all events from the logs so that leads me to believe that my DEST_KEY and FORMAT or configured correctly.

Why isn't this filtering events and forwarding to my indexers?

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

somesoni2 — Wed, 22 Feb 2017 17:05:55 GMT

Here is the documentation for 'Keep specific events and discard the rest'. Comparing that with your configuration, I would say everything looks good syntax wise. So, one thing that may cause issues is that REGEX may be a problem. So, does your raw data is contains that specific IP address (as content, not as host)??

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

merrelr — Wed, 22 Feb 2017 17:33:00 GMT

I'll doublecheck the regex. I've tried just making a really generic regex that match most events.

Does the regex need to match the whole entry or does the regex just need to be in the entry.

Would the following work?
REGEX = 192.168.99.2

2017-02-22T13:40:10+00:00 TS-C-ASA5585-L2L : %ASA-6-302013: Built inbound TCP connection 1299791138 for rav:192.168.99.2/26132 (192.168.99.2/26132) to erv:192.168.214.56/829 (192.168.214.56/829)

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

jkat54 — Wed, 22 Feb 2017 17:40:30 GMT

This matches every character:

 [setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

So in effect you're sending every event to nullQueue if you execute the setnull transform prior to the other.

Try this instead in props.conf:

 [firewall]
 TRANSFORMS-AAA = setparsing
 TRANSFOMRS-zzz = setnull

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

merrelr — Tue, 29 Sep 2020 12:57:37 GMT

Splunk shows to set it up that way.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

mrgibbon — Wed, 22 Feb 2017 22:28:28 GMT

This is an example I have used before, you are correct the null goes first:

    transforms.conf
    [trash]
    REGEX= .
    DEST_KEY = queue
    FORMAT = nullQueue

    [whitelist]
    REGEX = type=PATH|type=SYSCALL
    DEST_KEY = queue
    FORMAT = indexQueue

    [blacklist]
   REGEX = \/u0(1|2|3|4|5)\/blah\/|\/u01\/blah\/JDE_HOME\/logs|\/u01\/blah\/XXD_HOME\/data
   DEST_KEY = queue
   FORMAT = nullQueue

props.conf
[source::/var/log/audit/audit.log*]
TRANSFORMS-set = trash,whitelist,blacklist

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

aaraneta_splunk — Thu, 20 Apr 2017 01:02:36 GMT

@merrelr - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

Re: How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

merrelr — Fri, 21 Apr 2017 12:46:40 GMT

Figured out the solution to this problem.

Turns out we had another props file with a firewall stanza as well. The stanzas in your props files need to be unique.