Why did the search head crash?

[danielbb]

A search head crashed and the last message in the log file is -

-- 06-22-2020 11:13:39.341 -0400 WARN  PeriodicReapingTimeout - Spent 20995ms reaping search artifacts in /opt/apps/splunk/var/run/splunk/dispatch

What can it be?

[anilchaithu]

@danielbb 

The reap_time (time to clean up expired search artifacts) is 20995ms. This looks to be on higher side. Again this value is subjective to the setup.

This means you have more searches running at that time.

check for cpu and memory utilization for this SH at the same time.

[danielbb]

In the MC under Resource Usage: Instance, I see that the SH maxed out the memory. Where can I see the search(es) that caused it?

[anilchaithu]

@danielbb 

check here

MC -> search tab -> activity -> search activity : instance

the bottom panel will list top memory consuming searches

[danielbb]

Search > Activity > Instance shows the top 20 memory searches and nothing stands out.

Under Resource Usage: Instance, the graph shows us that **search** took 10 GBs of memory just before the crash and the **splunkd** process took over 20 GBs of memory. What can it be?

[anilchaithu]

@danielbb 

check splunkd.log for more errors leading to the crash.

If you still not able to figure out, open a support ticket with splunk and share diag.