Splunk Enterprise

Why won't 'collect' command work?

shob4726
Observer

Hey,
I have a Splunk Enterprise environment with servers cluster of 4 SHDs, 5 HFDs and 3 Indexers.
In addition there is a number of alerts that are configured on my Search Heads, the alerts use the 'collect' command which indexes the returned events
from the query to some index.
For example:
index=Example ... | collect index=production

It's worked for some time, approximately 6 months. But now, when I try to search for events on index "production", I get 0 events.
I searched for errors and bugs with the support of a Splunk specialist, but we didn't find a solution.

One speculation that we had was the 'stashParsing' queue which configured on the SHDs and used by the 'collect' command.
We found on the '_internal' index logs about the queue 'max_size=500KB' and 'current_size'.
The 'current_size' values were 0 99.9% of the time and 494, 449, 320, 256 0.001% of the time on the last 30 days.
I have tried increasing the 'max_size' of the queue
I have created a file named 'server.conf' in the following location: $SPLUNK_HOME/etc/shcluster/apps/shd_base.
The file content is:
[stashparsing]
maxsize=600MB
I have distributed this to the SHDs cluster, but it did not seem to have any effect.

Splunk version: 8.1.3
Linux version: Red Hat Linux Enterprise 7.8
This is an air-gapped environment so I cannot attach any logs or data.

0 Karma

PaulPanther
Motivator

The stanza in the server.conf is wrong.

Reconfigure it as

[queue=stashparsing]
maxSize=600MB

 Could you try to execute an adhoc search and use the collect command for that result? 

0 Karma

shob4726
Observer

I've fixed the syntax but unfortunately no behavior change.

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with Paul Stout

Welcome to our very first developer spotlight release series where we'll feature some awesome Splunk ...

State of Splunk Careers 2024: Maximizing Career Outcomes and the Continued Value of ...

For the past four years, Splunk has partnered with Enterprise Strategy Group to conduct a survey that gauges ...

Data-Driven Success: Splunk & Financial Services

Splunk streamlines the process of extracting insights from large volumes of data. In this fast-paced world, ...