How to edit props.conf and transforms.conf on a heavy forwarder to keep specific events and discard the rest

[merrelr]

props.conf

[firewall]
TRANSFORMS-set = setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = 192\.168\.1\.1
DEST_KEY = queue
FORMAT = indexQueue

I have a heavy forwarder with the following. What I want to do is only forward events that match the regex to our indexers for indexing and discard the rest. It doesn't matter what put in the REGEX section though nothing comes through even if I look at the logs and see that there are definitely matches.

If I change props.conf to

TRANSFORMS-set = setparsing

I get all events from the logs so that leads me to believe that my DEST_KEY and FORMAT or configured correctly.

Why isn't this filtering events and forwarding to my indexers?

[merrelr · Accepted solution]

Figured out the solution to this problem.

Turns out we had another props file with a firewall stanza as well. The stanzas in your props files need to be unique.

[aaraneta_splunk]

@merrelr - Did one of the answers below help provide a solution your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

[mrgibbon]

This is an example I have used before, you are correct the null goes first:

    transforms.conf
    [trash]
    REGEX= .
    DEST_KEY = queue
    FORMAT = nullQueue

    [whitelist]
    REGEX = type=PATH|type=SYSCALL
    DEST_KEY = queue
    FORMAT = indexQueue

    [blacklist]
   REGEX = \/u0(1|2|3|4|5)\/blah\/|\/u01\/blah\/JDE_HOME\/logs|\/u01\/blah\/XXD_HOME\/data
   DEST_KEY = queue
   FORMAT = nullQueue

props.conf
[source::/var/log/audit/audit.log*]
TRANSFORMS-set = trash,whitelist,blacklist

[jkat54]

This matches every character:

 [setnull]
 REGEX = .
 DEST_KEY = queue
 FORMAT = nullQueue

So in effect you're sending every event to nullQueue if you execute the setnull transform prior to the other.

Try this instead in props.conf:

 [firewall]
 TRANSFORMS-AAA = setparsing
 TRANSFOMRS-zzz = setnull

[merrelr]

Splunk shows to set it up that way.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Keep_specific_event...

[somesoni2]

Here is the documentation for 'Keep specific events and discard the rest'. Comparing that with your configuration, I would say everything looks good syntax wise. So, one thing that may cause issues is that REGEX may be a problem. So, does your raw data is contains that specific IP address (as content, not as host)??

http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Keep_specific_event...

[merrelr]

I'll doublecheck the regex. I've tried just making a really generic regex that match most events.

Does the regex need to match the whole entry or does the regex just need to be in the entry.

Would the following work?
REGEX = 192.168.99.2

2017-02-22T13:40:10+00:00 TS-C-ASA5585-L2L : %ASA-6-302013: Built inbound TCP connection 1299791138 for rav:192.168.99.2/26132 (192.168.99.2/26132) to erv:192.168.214.56/829 (192.168.214.56/829)