Why won't 'collect' command work?

shob4726 — Tue, 17 Jan 2023 20:16:55 GMT

Hey,
I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers.
In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events
from the query to some index.
For example:
index=Example ... | collect index=production

It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events.
I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.

One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command.
We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'.
The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days.
I have tried increasing the 'max_size' of the queue
I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base.
The file content is:
[stashparsing]
maxsize=600MB
I have distributed this to the SHDs cluster, but it did not seem to have any effect.

Splunk version: 8.1.3
Linux version: Red Hat Linux Enterprise 7.8
This is an air-gapped environment so I cannot attach any logs or data.

Re: 'collect' command doesn't work

PaulPanther — Mon, 16 Jan 2023 15:19:13 GMT

The stanza in the server.conf is wrong.

Reconfigure it as

[queue=stashparsing] maxSize=600MB

Could you try to execute an adhoc search and use the collect command for that result?

Re: 'collect' command doesn't work

shob4726 — Tue, 17 Jan 2023 08:31:25 GMT

I've fixed the syntax but unfortunately no behavior change.

topic Re: 'collect' command doesn't work in Splunk Enterprise

Why won't 'collect' command work?

Re: 'collect' command doesn't work

Re: 'collect' command doesn't work