topic Re: Need help with nullQueue (specifics included) in Splunk Search

Need help with nullQueue (specifics included)

echojacques — Mon, 28 Sep 2020 14:36:22 GMT

Hi everyone,

For a few days now I've been tweaking my props.conf, transforms.conf, and rebooting Splunk trying to exclude certain events from being indexed (nullQueue). I have included the stanzas in my props.conf, transforms.conf, and an example of an event that I am trying to exclude below. I'm hoping that someone can save me another week of tweaking and rebooting Splunk trying to get this nullQueue to work 🙂

My app-local-props.conf file contains:

[source::*opsec*]
TRANSFORMS-null:setnull

Note: I'm not sure if I identifed the source correctly, you can see the full long path in the example event I included below.

My app-local-transforms.conf file contains:

[setnull]
REGEX=(?m)^service=(80)
DEST_KEY=queue
FORMAT=nullQueue

What I'm trying to exclude:

Here's an example of one of the multi-line events that contain "service=80" that I'm trying to send to the nullQueue. I modified the original event to shorten the length and also changed the IP addresses. You can see "service=80" near the middle of the second line:

I've tried many different variations (at least 20) of the REGEX but nothing has worked so far. Any advice or guidance is very appreciated!!

Thanks

Re: Need help with nullQueue (specifics included)

Ayn — Mon, 19 Aug 2013 21:04:53 GMT

I can see a couple of problems.

First and most importantly, your regex is looking for "service=80" at the start of the line. (That's the significance of ^). So your regex will not match your sample event. I would advise you to try your regexes in a tool like http://regexpal.com/ or http://www.gskinner.com/RegExr/ so you know that they match correctly.

Also, you should make sure that the source stanza ([source::*opsec*]) is being identified correctly. Wildcards will match anything but slashes so if you do have slashes in your source, chances are your settings for this source aren't being applied at all.

Finally you have "TRANSFORMS-null:setnull" - there should not be a colon there, it should be an equals sign, so "TRANSFORMS-null = setnull".

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 21:22:05 GMT

Ayn,
Thanks for the help. I used regexpal and modified my files as follows:

props.conf (modified source and fixed colon):

[source::(/data/splunk/etc/apps/Splunk_TA_opseclea_linux22/bin/lea-loggrabber.sh --configentity SplunkLEA)]
 TRANSFORMS-null=setnull

transforms.conf (simplified REGEX per regexpal):

[setnull]
REGEX=service=80
DEST_KEY=queue
FORMAT=nullQueue

I am rebooting Splunk right now and will let you know if it works!

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 21:27:48 GMT

Still not working... should I use quotes around the source in the props source instead of parens since it's a long source with hyphens?

Re: Need help with nullQueue (specifics included)

Ayn — Mon, 19 Aug 2013 21:30:48 GMT

That source looks really weird, so it's very likely that it's not matching properly. Is that really the exact source you're getting for your opsec data in Splunk? For troubleshooting purposes I'd throw in a field extraction that is guaranteed to match something under the same stanza to see if it's being applied or not. That way you don't have to go restart Splunk to see if the stanza is being applied or not.

Re: Need help with nullQueue (specifics included)

Ayn — Mon, 19 Aug 2013 21:31:41 GMT

Also, apart from this - just to make sure - you are putting all this configuration on the indexer, not a forwarder, I hope.

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 21:54:59 GMT

Yes, it's all on the indexer (the Linux system that is running our Splunk). Yes, that's the exact source as it appears under "Sources" on the main Splunk search page. Maybe I should drop the "--configentity SplunkLEA" part which looks like a switch or a flag?

I was able to successfully exclude some windows events from the Splunk index last month so I know that the nullQueue works. Just can't get it to work for my opsec.

Also, I am modifying the files in the APP LOCAL directory, not in the APP DEFAULT directory. Hope this is correct.

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 21:57:28 GMT

Oh, and the opsec source is actually the OPSEC LEA add-on.

Re: Need help with nullQueue (specifics included)

kristian_kolb — Mon, 19 Aug 2013 22:02:18 GMT

Use your sourcetype instead of the source in props.conf, if possible.

and yes, do it in local, not in default. Just to ensure that it is going to happen, make the changes to props/transforms in $SPLUNK_HOME/etc/system/local

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 22:05:32 GMT

kristian,

So I will modify my props.conf to use the sourcetype like this:

[sourcetype::opsec]
 TRANSFORMS-null=setnull

I will reboot and let you know if it works. Also, I thought I had to make the changes in the APP directory??

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 22:14:24 GMT

Also, not sure if it matters, but I have the stanzas at the top/beginning of my props.conf and transforms.conf files.

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 22:16:03 GMT

Using sourcetype::opsec instead did not work...

Re: Need help with nullQueue (specifics included)

echojacques — Mon, 19 Aug 2013 22:34:33 GMT

I had to disable the app since I was getting close to exceeding my license due to this problem.

Will try again tomorrow.

Re: Need help with nullQueue (specifics included)

lukejadamec — Mon, 19 Aug 2013 23:36:24 GMT

Global stanzas should go at the top (best practice), correct stanzas work where they belong.
Kristian gave you the quick answer to this document: http://docs.splunk.com/Documentation/Splunk/4.3.4/Admin/Wheretofindtheconfigurationfiles
Which basically says that index time activity happens first at the global level, which is controlled first by the etc/system/local configuration files.

Re: Need help with nullQueue (specifics included)

sowings — Mon, 19 Aug 2013 23:46:53 GMT

Sourcetype definitions in props.conf don't use "sourcetype::" as a prefix. If your type is called "mytype" the stanza in props.conf would be [mytype].

Strictly speaking, I don't think you need the (?m) as events coming out of the LEA aren't multiline, as best as I can remember. Instead, they're simply long, and wrap in the UI.

I test my regexes for correctness in the UI before applying the transforms.conf rule to drop the events (making sure I got the right syntax before applying a potentially destructive rule). You could use regexr or other tools as Ayn suggests, but if you've already got similar events in Splunk and just want to validate the regex, try using the regex command:

sourcetype=opsec | regex _raw="service=80" This only returns rows where the whole line (_raw field) contains the service=80 string. If you get events back, then you know you've found the rule. But check the discovered service field, as this would match service=8000 or whatever as well. You might want to use the | as anchors in your regex, but remember that those would have to be escaped:

sourcetype=opsec | regex _raw="|service=80|". When you're satisfied that you have the regex right, cut and paste it into your transforms, restart your indexer, bingo.

Re: Need help with nullQueue (specifics included)

kristian_kolb — Tue, 20 Aug 2013 08:24:44 GMT

echojaques: if the sourcetype is opsec then the stanza in props.conf should be;

[opsec] TRANSFORMS-null = setnull

Doing it in etc/system/local just ensures that the setting will have the highest priority and will always be active, regardelss of whether you enable/disable certain apps.

Re: Need help with nullQueue (specifics included)

echojacques — Tue, 20 Aug 2013 15:02:03 GMT

lukejadamec: thanks for the explanation, that makes sense. I'll test it at the global level today.

kristian: i have changed my props stanza to [opsec]. I'll let you know if it works after rebooting and testing.

I have another nullQueue (that works) setup for my Windows WMI data. It is also using: TRANSFORMS-null = setnull. Is it OK that it has the same name (setnull) or should I change this nullQueue name to setnull-opsec?

Thanks!

Re: Need help with nullQueue (specifics included)

echojacques — Tue, 20 Aug 2013 15:08:27 GMT

sowings: Thanks very much for all of this information. I have removed (?m) from my REGEX and I'm now testing this REGEX in my transforms:

[setnull]
REGEX="service=80"

I'll use the | anchor to limit to "80" once I get "service=80" to work (trying to keep it simple at first).

I did test my regex in regexpal.com and also in the search UI before modifying it in transforms. But I wasn't using the "regex_raw" command before (I was just typing "service=80" etc. into the UI search) so thanks for that tip.

I'll test again this morning and let you know if it works.
Thanks!

Re: Need help with nullQueue (specifics included)

echojacques — Tue, 20 Aug 2013 15:45:35 GMT

Unfortunately it's still not working. I modified my props stanza to [opsec] and used the simple REGEX="service=80". I have also modified the props.conf and transforms.conf in the global directory instead of the app directory and not working either. I'm all out of ideas on this... might need to call for Splunk support.

Re: Need help with nullQueue (specifics included)

echojacques — Tue, 20 Aug 2013 15:46:25 GMT

Ok, I have "moved" the nullQueue from the APP-local directory to the home-etc-system-local directory and still not working...