Hey,
I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers.
In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events
from the query to some index.
For example:
index=Example ... | collect index=production
It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events.
I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.
One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command.
We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'.
The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days.
I have tried increasing the 'max_size' of the queue
I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base.
The file content is:
[stashparsing]
maxsize=600MB
I have distributed this to the SHDs cluster, but it did not seem to have any effect.
Splunk version: 8.1.3
Linux version: Red Hat Linux Enterprise 7.8
This is an air-gapped environment so I cannot attach any logs or data.
