Blacklist log events (not log filenames) using a string to limit events forwarding to Splunk Indexers

[ejwade]

I have a dns log that is very chatty with internal requests (e.g. localserver5.internal). I would like to forward dns logs for external requests (maliciouswebsite.g.mail.com) but filter out (blacklist) local requests. I have found ways to blacklist log files, but not specific log entries.

Another note - the field extraction for this value occurs at the search head, not the app with the inputs file. For the host with the dns log, I created a custom app in our deployment server with a custom inputs.conf for monitoring this log. Not sure if that affects what's possible.

[damien_chillet]

Hi ejwade,

You can send specific events to the nullQueue to discard them at the indexer/heavy forwarder level.

In your case it would look like:

props.conf

[source::<bro_logs_source>]
TRANSFORMS-null= set null

transfroms.conf

[setnull]
REGEX = <your_regex> (for you something that deals with internal A record)
DEST_KEY = queue
FORMAT = nullQueue

You can have a read through the "Filter event data and send to queues" section at http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad.

Hope that's helpful!

[sshelly_splunk]

Can upload some anonymized data ? This would occur at the indexer or heavy forwarder via regex most likely. A sample log (with more than a few whitelist and blacklist) events should suffice.

[ejwade]

Sure. These are bro DNS logs, so they are tab delimited (I'll do comma below).

1511991992.963051,CE0oKO1yiHQLlxOB5g,10.10.10.10,47041,10.20.20.20,53,udp,13336,internal-srv.ewade.internal,C_INTERNET,1,A,0,NOERROR,T,F
1511991994.963051,CE0oKO1yweQLlxOB5g,10.10.10.10,47041,10.20.20.20,53,udp,13336,maliciouswebsite.g.mail.com,C_INTERNET,1,A,0,NOERROR,T,F

internal-srv.ewade.internal is the "A" record that we want to filter out, while maliciouswebsite.g.mail.com is the one we want to pass to Splunk. A RegEx would suffice, but I'm not sure where to do this or the syntax. "blacklist" under inputs.conf seems to only refer to filenames.