Getting Data In

Unable to distribute to peer oddness

tgiles
Path Finder

Hi,

I have two pooled search heads which search a couple of indexers. heads connect across a public IP address to the indexers. The indexers have a private IP address name in their configuration. For example:

  • Indexer 1: peer name 192.0.32.10:8089, splunk server name 10.999.20.5
  • Indexer 2: peer name 192.0.32.11:8089, splunk server name 10.999.20.6

intermittently, a search head will throw an error bar at the top, reporting a connection problem:

Unable to distribute to peer named
192.0.32.10:8089 at uri https://192.0.32.10:8089 because peer
has status = "Down".

The message is confusing because it's giving the 'peer name' as the wrong thing. If it was an actual error, i'd assume it would call the problem peer by the correct peer name (in this instance, 10.999.20.5) and not the external IP.

I wrote up a little python scripts to constantly make socket network connections to the affected indexer. Even when Splunk reported it couldn't connect to the indexer, my script had no issues opening network connections.

Double-checked all the splunk indexer configuration files, just in case I have a bad configuration somewhere- everything looks clean. networking guys reported no issues with the firewall logs. Indexer itself looks fine, logs are coming in, no problems observed in the logs on it.

have any ideas on how to troubleshoot an indexer connection problem when I can't replicate it using another method? any particulars to look for in the logs?

Thanks for your input

Tags (3)

lmyrefelt
Builder

Well its hard to say without having a look at your confs. But i "useally" get this if the indexer is under heavy load and can responed to all requests. When in searchhead pooling the performance of the central located share (nfs/cifs/ whatever) is important as well since the results and whatnot is saved if not only temporary there.

For multihome splunk (indexer) instanses there seems to be some things you can look at.

$SPLUNK_HOME/etc/splunk-launch.conf
SPLUNK_BINDIP=

$SPLUNK_HOME/etc/system/local/web.conf
mgmtHostPort =
server.socket_host =

0 Karma

lmyrefelt
Builder
0 Karma

lukereeves
Engager

This is happening to me as well, I guess you never found a resolution? I'll make a support ticket for this.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...