Hey, I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers. In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events from the query to some index. For example: index=Example ... | collect index=production
It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events. I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.
One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command. We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'. The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days. I have tried increasing the 'max_size' of the queue I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base. The file content is: [stashparsing] maxsize=600MB I have distributed this to the SHDs cluster, but it did not seem to have any effect.
Splunk version: 8.1.3 Linux version: Red Hat Linux Enterprise 7.8 This is an air-gapped environment so I cannot attach any logs or data.
... View more