Unable to Distribute

[tyronetv]

I have to identical servers. One acts as an indexing server and one as a user access search portal.

I am constantly getting reports from my users (all 15 of them so far) that they are getting the following yellow band warning:

Unable to distribute to peer named [machine name:port] at uri https://[machine name:port] because peer has status = "Down."

When, in fact, I can log into the indexing machine and find it running normally. If refresh is hit often enough (sometimes 20 - 30 times) on the search head the search will go through. OR, if they get me to restart splunk on the search head it will sometimes clear.

The problem here is that 'sometimes' is not 'all the time.' I, myself, have hit the search button over 30 times and still the warning persists and the search fails.

Any ideas on why this happens? The network is clean and clear between the two data centers. And all other activity across the wan is operating normal with no errors so I think it might be my indexing machine.

What can I provide to assist in the understanding? (Both for myself and others kind enough to jump in). 🙂

[lguinn2 · Accepted solution]

This may not be an answer to your question, sorry. But generally, if you have only 2 Splunk servers, both of them should be indexers. One search head + one indexer is not usually a recommended configuration. Read the sections "Ratio of indexers to search heads" and "Accommodating many simultaneous searches" in the Installation Manual at http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdepl...

It is possible that the indexer is completely overloaded with searches and is therefore not responding. The search head then thinks that the indexer is "down." Making both servers into indexers will probably speed up your searches and reduce the overall load on the systems.

When you login to the indexing machine, what is it doing? If SplunkWeb is running on the indexer, can you login as the administrator and look at some of the standard dashboards regarding Indexing and Search? Can you look at Splunk's internal logs (index=_internal) and see if there are errors?

Finally, you might consider installing the SOS (Splunk on Splunk) app which includes some diagnostics that may help you understand what's going on with your indexer.

Good luck!

[kristian_kolb]

Do you have all 15 users logged in at the same time running searches? Does the problem occur only at peak usage? Does your search head have enough hardware to support the number of users? (I'm thinking this could be a resource shortage leading to timeouts). /k