Splunk Enterprise

Why won't 'collect' command work?

shob4726
Observer

Hey,
I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers.
In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events
from the query to some index.
For example:
index=Example ... | collect index=production

It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events.
I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.

One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command.
We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'.
The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days.
I have tried increasing the 'max_size' of the queue
I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base.
The file content is:
[stashparsing]
maxsize=600MB
I have distributed this to the SHDs cluster, but it did not seem to have any effect.

Splunk version: 8.1.3
Linux version: Red Hat Linux Enterprise 7.8
This is an air-gapped environment so I cannot attach any logs or data.

0 Karma

PaulPanther
Builder

The stanza in the server.conf is wrong.

Reconfigure it as

[queue=stashparsing]
maxSize=600MB

 Could you try to execute an adhoc search and use the collect command for that result? 

0 Karma

shob4726
Observer

I've fixed the syntax but unfortunately no behavior change.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...