topic Re: nullQueue not working in Getting Data In

nullQueue not working

wgawhh5hbnht — Fri, 13 Aug 2021 12:56:48 GMT

Here is my setup.

inputs.conf:

[script://./bin/lsof.sh]
interval = 600
sourcetype = lsof
source = lsof

props.conf:
[script://./bin/lsof.sh]

#also tried[lsof] & [source::lsof]

TRANSFORMS-null = null_splunk_user, null_splunk_command, null_splunk, lsof_normal_queue

transforms.conf:
[null_splunk_user]
REGEX = ^\S+\W+\d+\W+splunk\W+
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk_command]
REGEX = ^splunkd\W+\d+\W+splunk
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk]
REGEX = ^splunkd
DEST_KEY = queue
FORMAT = nullQueue

[lsof_normal_queue]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

sample of data:

splunkd 52507 splunk cwd DIR 202,1 4096 2 /
splunkd 52507 splunk rtd DIR 202,1 4096 2 /
splunkd 52507 splunk txt REG 202,1 76073192 409182 /opt/splunk/bin/splunkd

python2.7 53347 splunk cwd DIR 202,1 4096 2 /
splunk 53347 splunk rtd DIR 202,1 4096 2 /
splunk 53347 splunk txt REG 202,1 577688 411002 /opt/splunk/bin/splunk

splunkd 887 root cwd DIR 259,1 4096 2 /
splunkd 887 root rtd DIR 259,1 4096 2 /
splunkd 887 root txt REG 259,1 76073192 401488 /opt/splunk/bin/splunkd

On the indexer you can see that the props & transforms rules:

/opt/splunk/bin/splunk cmd btool props list --debug | grep lsof
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/props.conf [lsof]

/opt/splunk/bin/splunk cmd btool transforms list --debug | grep null_splunk
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk]
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk_command]
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk_user]

/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [lsof_normal_queue]

I've tried multiple iterations of regexes/props/transforms. I've been restarting the index clusters after each update to no avail. The majority of the data I'm attempting to drop is on the indexers themselves, splunk monitoring splunk.

Re: nullQueue not working

richgalloway — Wed, 11 Aug 2021 16:41:43 GMT

Have you tested your regular expression against your data samples in regex101.com?

BTW, btool does NOT show whether a props or transforms rule matches anything. It merely displays the current on-disk configuration.

Re: nullQueue not working

wgawhh5hbnht — Wed, 11 Aug 2021 17:22:27 GMT

Yup, here's all three of them together:

Re: nullQueue not working

richgalloway — Wed, 11 Aug 2021 18:35:24 GMT

It's interesting that line 2 looks like it should match, but isn't flagged as a match.

Does the data pass through a heavy forwarder? If so, then the props and transforms must be installed there.

Re: nullQueue not working

wgawhh5hbnht — Wed, 11 Aug 2021 18:47:19 GMT

Nope, this is data from the indexers being sent to the indexers.

Re: nullQueue not working

richgalloway — Wed, 11 Aug 2021 19:14:10 GMT

I'm not sure what you mean by "from the indexers being sent to the indexers".

Re: nullQueue not working

wgawhh5hbnht — Wed, 11 Aug 2021 19:21:40 GMT

the data is from the indexer cluster and it is being set to that same index cluster.

Re: nullQueue not working

richgalloway — Wed, 11 Aug 2021 20:21:31 GMT

I understand the words, but not the concept. Why is data being sent from an indexer cluster back into itself? What problem does that solve?

If this is "cooked" data then that would explain why nullQueue is not working. The data's been processed so Splunk won't process it again.

Re: nullQueue not working

wgawhh5hbnht — Thu, 12 Aug 2021 11:31:25 GMT

Regulations require that we monitor and alert on all activity on our splunk environment, including the indexers. Since we run splunk on linux, Splunk_TA_nix was the solution we came up with. We're getting the data in but now unfortunately we're over our license. I'm attempting to drop some events (noise) just to get us down below our license.

I'm not finding anything on how to filter cooked data, or for that fact why the data is cooked on the indexer. Maybe you could help me out and to make sure we're on the same page.

The inputs.conf is on splunkindexer01, as well as the props.conf and the transforms.conf. Shouldn't the data that is being ingested on splunkindexer01 go through the normal filtering process before it is sent to the index cluster? All the other props and transforms are working, why not the nullQueue?

Re: nullQueue not working

richgalloway — Thu, 12 Aug 2021 12:45:50 GMT

Got it. That sounds like the normal case of Splunk monitoring itself. That's not cooked data, just normal ingest.

IME, props that use nullQueue transforms also have a transform that sends the remaining data to an index.

Re: nullQueue not working

wgawhh5hbnht — Thu, 12 Aug 2021 14:01:19 GMT

Updated my props and transforms (see below) and restarted splunk service on indexers and still getting events from splunk (user) and splunkd (command).

props.conf:

[lsof]
TRANSFORMS-null = lsof_normal_queue, null_splunk_user, null_splunk_command, null_splunk

transforms.conf:

[null_splunk_user]
REGEX = (?m)^\S+\W+\d+\W+splunk\W+
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk_command]
REGEX = (?m)^splunkd\W+\d+\W+splunk
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk]
REGEX = (?m)^splunkd
DEST_KEY = queue
FORMAT = nullQueue

[lsof_normal_queue]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

Re: nullQueue not working

richgalloway — Thu, 12 Aug 2021 14:21:40 GMT

Recall that transforms are processed in the order given on the TRANSFORMS line in props.conf. That means lsof_normal_queue runs first, matches everything, and sends it to be indexed.

Re: nullQueue not working

wgawhh5hbnht — Thu, 12 Aug 2021 15:26:32 GMT

updated props, same outcome.

[lsof]
TRANSFORMS-null = null_splunk_user, null_splunk_command, null_splunk, lsof_normal_queue

Re: nullQueue not working

richgalloway — Thu, 12 Aug 2021 15:28:23 GMT

Now I'm officially stumped. Sorry.

Re: nullQueue not working

scelikok — Fri, 13 Aug 2021 04:56:33 GMT

Hİ @wgawhh5hbnht,

Can you try with below props.conf? Sometimes catching data from sourcetype not working.

[source::lsof] TRANSFORMS-null = lsof_normal_queue, null_splunk_user, null_splunk_command, null_splunk

Re: nullQueue not working

wgawhh5hbnht — Fri, 13 Aug 2021 12:58:29 GMT

Neither [source://lsof] nor [script://./bin/lsof.sh] worked either. After pushing the cluster I also did a rolling restart to make sure, still getting both splunk user & splunkd command logs.