- Splunk Answers
- :
- Splunk Platform Products
- :
- Splunk Enterprise
- :
- Re: Why did the search head crash?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Why did the search head crash?
A search head crashed and the last message in the log file is -
-- 06-22-2020 11:13:39.341 -0400 WARN PeriodicReapingTimeout - Spent 20995ms reaping search artifacts in /opt/apps/splunk/var/run/splunk/dispatch
What can it be?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The reap_time (time to clean up expired search artifacts) is 20995ms. This looks to be on higher side. Again this value is subjective to the setup.
This means you have more searches running at that time.
check for cpu and memory utilization for this SH at the same time.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the MC under Resource Usage: Instance, I see that the SH maxed out the memory. Where can I see the search(es) that caused it?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
check here
MC -> search tab -> activity -> search activity : instance
the bottom panel will list top memory consuming searches
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Search > Activity > Instance shows the top 20 memory searches and nothing stands out.
Under Resource Usage: Instance, the graph shows us that **search** took 10 GBs of memory just before the crash and the **splunkd** process took over 20 GBs of memory. What can it be?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
check splunkd.log for more errors leading to the crash.
If you still not able to figure out, open a support ticket with splunk and share diag.
Extending Observability Content to Splunk Cloud
More Control Over Your Monitoring Costs with Archived Metrics!
New in Observability Cloud - Explicit Bucket Histograms
