Deployment Architecture

search head captain clsoing the connection cluster mamber

New Member

we are having 3 search head and they are in cluster. splunkd process went down in two search head. when i checked the master search head i see below mentioned error. i restarted splunkd issue got resolved but need to understand why this was happened as all scheduled searches and reports were failed during that duration.

WARN TcpOutputFd - Connect to :8999 failed. Connection refused
ERROR TcpOutputFd - Connection to host=xxxx:8999 failed
WARN ArtifactReplicator - Connection failed

Tags (2)
0 Karma

Path Finder

If 2 out of 3 SH nodes were down, the your cluster was not "working". So the SH Captain was not able to reach the other Search Heads which can cause connection refused errors.

Did you have the error after starting the 2 nodes that were down?

You can see your cluster's running status by running this command on any SH cluster member.
$SPLUNK_HOME/bin/splunk show shcluster-status

0 Karma

SplunkTrust
SplunkTrust

Do you have captain_is_adhoc_searchhead = true in your server.conf? If so, then only the non-captain search heads will run scheduled searches.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

New Member

we do not have that attributes defined in our server.conf...

0 Karma

New Member

how it works without that

0 Karma