Unable to distribute to peer oddness

[tgiles]

Hi,

I have two pooled search heads which search a couple of indexers. heads connect across a public IP address to the indexers. The indexers have a private IP address name in their configuration. For example:

• Indexer 1: peer name 192.0.32.10:8089, splunk server name 10.999.20.5
• Indexer 2: peer name 192.0.32.11:8089, splunk server name 10.999.20.6

intermittently, a search head will throw an error bar at the top, reporting a connection problem:

Unable to distribute to peer named
192.0.32.10:8089 at uri https://192.0.32.10:8089 because peer
has status = "Down".

The message is confusing because it's giving the 'peer name' as the wrong thing. If it was an actual error, i'd assume it would call the problem peer by the correct peer name (in this instance, 10.999.20.5) and not the external IP.

I wrote up a little python scripts to constantly make socket network connections to the affected indexer. Even when Splunk reported it couldn't connect to the indexer, my script had no issues opening network connections.

Double-checked all the splunk indexer configuration files, just in case I have a bad configuration somewhere- everything looks clean. networking guys reported no issues with the firewall logs. Indexer itself looks fine, logs are coming in, no problems observed in the logs on it.

have any ideas on how to troubleshoot an indexer connection problem when I can't replicate it using another method? any particulars to look for in the logs?

Thanks for your input

[lmyrefelt]

Well its hard to say without having a look at your confs. But i "useally" get this if the indexer is under heavy load and can responed to all requests. When in searchhead pooling the performance of the central located share (nfs/cifs/ whatever) is important as well since the results and whatnot is saved if not only temporary there.

For multihome splunk (indexer) instanses there seems to be some things you can look at.

$SPLUNK_HOME/etc/splunk-launch.conf
SPLUNK_BINDIP=

$SPLUNK_HOME/etc/system/local/web.conf
mgmtHostPort =
server.socket_host =

[lmyrefelt]

http://splunk-base.splunk.com/answers/97606/search-peer-status-keeps-changing-every-minute-why

can give some more hints 🙂

[lukereeves]

This is happening to me as well, I guess you never found a resolution? I'll make a support ticket for this.