Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- SHC performance issue
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
SHC performance issue
Hi ,
We have 3 search heads in our search head clustered environment,
first we saw Raft issues in the captain, then we had referred this document : https://docs.splunk.com/Documentation/Splunk/7.2.3/DistSearch/Handleraftissues even after that we are facing same issues like the captain is going down first and later on the members one by one.
We have also found one more error "child killed by signal 9"
Also there is some false error messages appearing in DMC regarding the shc is down, bring it back online ASAP to avoid service disruption.
Can anyone please help with resolving these issues as the search heads are going down abruptly?
Thanks in Advance!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You need to increase memory on your SHC nodes, increase ulimit settings, or reduce memory consumption by Splunk.
When a process is killed by signal 9 that indicates that the kernel is protecting itself by killing processes that are consuming or reserving more memory than is available or allowed.
An upvote would be appreciated and Accept Solution if it helps!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
seems like a crash: child killed by signal 9 you'll need to reach out for support as this could be due to a bug. You're on 7.2.3 ?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
SHC is a pretty complicated setup, and it's very finicky. If you're not familiar with operating and maintaining a SHC, it might be best to get Splunk Professional Services involved, or open a support ticket for your issue.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Now that is not easy to troubleshoot without more information about your environment.
- What are the specs of the SHs?
- Are you running a lot of schedules searches? What's happening before they go down? Please take a look into the Monitoring Console (I suggest using it, it gives very good insight especially for troubleshooting issues) and analyze the SHC dashboards provided there.
- what was the reason the processes got killed? Memory problems maybe?
Skalli
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
[Puzzles] Solve, Learn, Repeat: Matching cron expressions
Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass
May 2026 Splunk Expert Sessions: Security & Observability
