nullQueue not working

[wgawhh5hbnht]

Here is my setup.

inputs.conf:

[script://./bin/lsof.sh]
interval = 600
sourcetype = lsof
source = lsof

props.conf:
[script://./bin/lsof.sh]

#also tried[lsof] & [source::lsof]

TRANSFORMS-null = null_splunk_user, null_splunk_command, null_splunk, lsof_normal_queue

transforms.conf:
[null_splunk_user]
REGEX = ^\S+\W+\d+\W+splunk\W+
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk_command]
REGEX = ^splunkd\W+\d+\W+splunk
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk]
REGEX = ^splunkd
DEST_KEY = queue
FORMAT = nullQueue

[lsof_normal_queue]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

sample of data:

splunkd 52507 splunk cwd DIR 202,1 4096 2 /
splunkd 52507 splunk rtd DIR 202,1 4096 2 /
splunkd 52507 splunk txt REG 202,1 76073192 409182 /opt/splunk/bin/splunkd

python2.7 53347 splunk cwd DIR 202,1 4096 2 /
splunk 53347 splunk rtd DIR 202,1 4096 2 /
splunk 53347 splunk txt REG 202,1 577688 411002 /opt/splunk/bin/splunk

splunkd 887 root cwd DIR 259,1 4096 2 /
splunkd 887 root rtd DIR 259,1 4096 2 /
splunkd 887 root txt REG 259,1 76073192 401488 /opt/splunk/bin/splunkd

On the indexer you can see that the props & transforms rules:

/opt/splunk/bin/splunk cmd btool props list --debug | grep lsof
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/props.conf [lsof]

/opt/splunk/bin/splunk cmd btool transforms list --debug | grep null_splunk
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk]
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk_command]
/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [null_splunk_user]

/opt/splunk/etc/slave-apps/Splunk_TA_nix/local/transforms.conf [lsof_normal_queue]

I've tried multiple iterations of regexes/props/transforms. I've been restarting the index clusters after each update to no avail. The majority  of the data I'm attempting to drop is on the indexers themselves, splunk monitoring splunk.

[scelikok]

Hİ @wgawhh5hbnht,

Can you try with below props.conf? Sometimes catching data from sourcetype not working.

[source::lsof]
TRANSFORMS-null = lsof_normal_queue, null_splunk_user, null_splunk_command, null_splunk

[wgawhh5hbnht]

Neither [source://lsof] nor [script://./bin/lsof.sh] worked either. After pushing the cluster I also did a rolling restart to make sure, still getting both splunk user & splunkd command logs.

[richgalloway]

Have you tested your regular expression against your data samples in regex101.com?

BTW, btool does NOT show whether a props or transforms rule matches anything.  It merely displays the current on-disk configuration.

[wgawhh5hbnht]

Yup, here's all three of them together:

[richgalloway]

It's interesting that line 2 looks like it should match, but isn't flagged as a match.

Does the data pass through a heavy forwarder?  If so, then the props and transforms must be installed there.

[wgawhh5hbnht]

Nope, this is data from the indexers being sent to the indexers.

[richgalloway]

I'm not sure what you mean by "from the indexers being sent to the indexers".

[wgawhh5hbnht]

the data is from the indexer cluster and it is being set to that same index cluster.

[richgalloway]

I understand the words, but not the concept.  Why is data being sent from an indexer cluster back into itself?  What problem does that solve?

If this is "cooked" data then that would explain why nullQueue is not working.  The data's been processed so Splunk won't process it again.

[wgawhh5hbnht]

Regulations require that we monitor and alert on all activity on our splunk environment, including the indexers. Since we run splunk on linux, Splunk_TA_nix was the solution we came up with. We're getting the data in but now unfortunately we're over our license. I'm attempting to drop some events (noise) just to get us down below our license.

I'm not finding anything on how to filter cooked data, or for that fact why the data is cooked on the indexer. Maybe you could help me out and to make sure we're on the same page.

The inputs.conf is on splunkindexer01, as well as the props.conf and the transforms.conf. Shouldn't the data that is being ingested on splunkindexer01 go through the normal filtering process before it is sent to the index cluster? All the other props and transforms are working, why not the nullQueue?

[richgalloway]

Got it. That sounds like the normal case of Splunk monitoring itself.  That's not cooked data, just normal ingest.

IME, props that use nullQueue transforms also have a transform that sends the remaining data to an index.

[wgawhh5hbnht]

Updated my props and transforms (see below) and restarted splunk service on indexers and still getting events from splunk (user) and splunkd (command).

props.conf:

[lsof]
TRANSFORMS-null = lsof_normal_queue, null_splunk_user, null_splunk_command, null_splunk

transforms.conf:

[null_splunk_user]
REGEX = (?m)^\S+\W+\d+\W+splunk\W+
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk_command]
REGEX = (?m)^splunkd\W+\d+\W+splunk
DEST_KEY = queue
FORMAT = nullQueue

[null_splunk]
REGEX = (?m)^splunkd
DEST_KEY = queue
FORMAT = nullQueue

[lsof_normal_queue]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

[richgalloway]

Recall that transforms are processed in the order given on the TRANSFORMS line in props.conf.  That means lsof_normal_queue runs first, matches everything, and sends it to be indexed.

[wgawhh5hbnht]

updated props, same outcome.

[lsof]
TRANSFORMS-null = null_splunk_user, null_splunk_command, null_splunk, lsof_normal_queue

[richgalloway]

Now I'm officially stumped.  Sorry.