Need help with nullQueue (specifics included)

[echojacques]

Hi everyone,

For a few days now I've been tweaking my props.conf, transforms.conf, and rebooting Splunk trying to exclude certain events from being indexed (nullQueue). I have included the stanzas in my props.conf, transforms.conf, and an example of an event that I am trying to exclude below. I'm hoping that someone can save me another week of tweaking and rebooting Splunk trying to get this nullQueue to work 🙂

My app-local-props.conf file contains:

[source::*opsec*]
TRANSFORMS-null:setnull

Note: I'm not sure if I identifed the source correctly, you can see the full long path in the example event I included below.

My app-local-transforms.conf file contains:

[setnull]
REGEX=(?m)^service=(80)
DEST_KEY=queue
FORMAT=nullQueue

What I'm trying to exclude:

Here's an example of one of the multi-line events that contain "service=80" that I'm trying to send to the nullQueue. I modified the original event to shorten the length and also changed the IP addresses. You can see "service=80" near the middle of the second line:

loc=12345|time=19Aug2013 14:44:28|action=drop|orig=10.10.10.10|i/f_dir=inbound|i/f_name=eth1-01|has_accounting=0|src=10.10.10.10|s_port=49528|dst=10.10.10.10|service=80|proto=tcp|message_info=Address spoofinghost=SPLUNK-01 | sourcetype=opsec | source=/data/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity SplunkLEA

I've tried many different variations (at least 20) of the REGEX but nothing has worked so far. Any advice or guidance is very appreciated!!

Thanks

[echojacques]

Yes, it's all on the indexer (the Linux system that is running our Splunk). Yes, that's the exact source as it appears under "Sources" on the main Splunk search page. Maybe I should drop the "--configentity SplunkLEA" part which looks like a switch or a flag?

I was able to successfully exclude some windows events from the Splunk index last month so I know that the nullQueue works. Just can't get it to work for my opsec.

Also, I am modifying the files in the APP LOCAL directory, not in the APP DEFAULT directory. Hope this is correct.

[Ayn]

Also, apart from this - just to make sure - you are putting all this configuration on the indexer, not a forwarder, I hope.

[Ayn]

That source looks really weird, so it's very likely that it's not matching properly. Is that really the exact source you're getting for your opsec data in Splunk? For troubleshooting purposes I'd throw in a field extraction that is guaranteed to match something under the same stanza to see if it's being applied or not. That way you don't have to go restart Splunk to see if the stanza is being applied or not.

[echojacques]

Still not working... should I use quotes around the source in the props source instead of parens since it's a long source with hyphens?

[echojacques]

Ayn,
Thanks for the help. I used regexpal and modified my files as follows:

props.conf (modified source and fixed colon):

[source::(/data/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity SplunkLEA)]
 TRANSFORMS-null=setnull

transforms.conf (simplified REGEX per regexpal):

[setnull]
REGEX=service=80
DEST_KEY=queue
FORMAT=nullQueue

I am rebooting Splunk right now and will let you know if it works!