Splunk Enterprise

Why won't 'collect' command work?

shob4726
Observer

Hey,
I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers.
In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events
from the query to some index.
For example:
index=Example ... | collect index=production

It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events.
I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.

One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command.
We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'.
The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days.
I have tried increasing the 'max_size' of the queue
I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base.
The file content is:
[stashparsing]
maxsize=600MB
I have distributed this to the SHDs cluster, but it did not seem to have any effect.

Splunk version: 8.1.3
Linux version: Red Hat Linux Enterprise 7.8
This is an air-gapped environment so I cannot attach any logs or data.

0 Karma

PaulPanther
Builder

The stanza in the server.conf is wrong.

Reconfigure it as

[queue=stashparsing]
maxSize=600MB

 Could you try to execute an adhoc search and use the collect command for that result? 

0 Karma

shob4726
Observer

I've fixed the syntax but unfortunately no behavior change.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Splunk Smartness with Brandon Sternfield | Episode 3

Hello and welcome to another episode of "Splunk Smartness," the interview series where we explore the power of ...

Monitoring Postgres with OpenTelemetry

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power ...

Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In ...