Splunk Enterprise

search head keeps restarting

dhanasekar79
New Member

Hi,


We have a Splunk distributed cluster setup with 3 indexers, 3 search heads, 1 cluster master.   The clusters were healthy when the cluster setup was done.

However, after switching the log traffic to Splunk indexer, we see one of the search head detached from the cluster and keeps restarting.

Below is the error I see from the splunkd.log for the search head that is having problem.

01-11-2022 17:39:57.495 +0000 INFO MetricSchemaProcessor [1424 typing] - channel confkey=source::/opt/splunk/var/log/introspection/disk_objects.log|host::splunk-shc-search-head-2|splunk_intro_disk_objects|CLONE_CHANNEL has an event with no measure, will be skipped.
01-11-2022 17:39:57.751 +0000 INFO IndexProcessor [1173 MainThread] - handleSignal : Disabling streaming searches.
01-11-2022 17:39:57.752 +0000 INFO IndexProcessor [1173 MainThread] - request state change from=RUN to=SHUTDOWN_SIGNALED
01-11-2022 17:39:57.752 +0000 INFO SHClusterMgr [1173 MainThread] - Starting to Signal shutdown RAFT
01-11-2022 17:39:57.752 +0000 INFO SHCRaftConsensus [1173 MainThread] - Shutdown signal received.
01-11-2022 17:39:57.752 +0000 INFO SHClusterMgr [1173 MainThread] - Signal shutdown RAFT completed
01-11-2022 17:39:57.752 +0000 INFO UiHttpListener [1173 MainThread] - Shutting down webui
01-11-2022 17:39:57.752 +0000 INFO UiHttpListener [1173 MainThread] - Shutting down webui completed

 

Any insights on what is causing this?

 

Labels (1)
Tags (1)
0 Karma

ayush1906
Path Finder

try restarting all the search heads 1 by 1, looks like it's unable to get in sync with all. 

 

and after this check, if the captain is being assigned and this search head took part in the election.

0 Karma

dhanasekar79
New Member

Thanks. I restarted multiple times and it doesn't sync-up.  As it is running in production, can't wait and had to delete the search head cluster and re-created it. It worked fine after. Still no clue why we had this problem for the first time.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

i haven’t been in this kind of situation, but in other cases remove that node from SHC and/or resync it from others has usually solved the situation. No need for recreate the whole SHC.

0 Karma
Get Updates on the Splunk Community!

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...