I have a dns log that is very chatty with internal requests (e.g. localserver5.internal). I would like to forward dns logs for external requests (maliciouswebsite.g.mail.com) but filter out (blacklist) local requests. I have found ways to blacklist log files, but not specific log entries.
Another note - the field extraction for this value occurs at the search head, not the app with the inputs file. For the host with the dns log, I created a custom app in our deployment server with a custom inputs.conf for monitoring this log. Not sure if that affects what's possible.
Can upload some anonymized data ? This would occur at the indexer or heavy forwarder via regex most likely. A sample log (with more than a few whitelist and blacklist) events should suffice.
Sure. These are bro DNS logs, so they are tab delimited (I'll do comma below).
internal-srv.ewade.internal is the "A" record that we want to filter out, while maliciouswebsite.g.mail.com is the one we want to pass to Splunk. A RegEx would suffice, but I'm not sure where to do this or the syntax. "blacklist" under inputs.conf seems to only refer to filenames.
You can send specific events to the nullQueue to discard them at the indexer/heavy forwarder level.
In your case it would look like:
[source::<bro_logs_source>] TRANSFORMS-null= set null
[setnull] REGEX = <your_regex> (for you something that deals with internal A record) DEST_KEY = queue FORMAT = nullQueue
You can have a read through the "Filter event data and send to queues" section at http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad.
Hope that's helpful!