Changes to forwarder->indexer connectivity in 6.3.0?

[emiller42]

Hello! I've upgraded my test environments to Splunk 6.3, and I'm noticing that my forwarders have begun throwing the following warnings in splunkd.log:

WARN  DistributedPeerManagerHeartbeat - Unable to get server info from peer: http://<IP OF INDEXER>:8089 due to: Connect Timeout; exceeded 10000 milliseconds

Now, I'm not sure why my forwarders are attempting to reach the management port on my indexers in the first place. These environments do not use index clustering, so the only port open between the two is 9997 for forwarding data. Is there some new functionality that needs the indexer management port to be accessible to the forwarder?

I'm not noticing any gaps in indexed data, so there doesn't appear to be any appreciable impact. But the warnings are concerning.

[amathew_splunk · Accepted solution]

Is the indexer set to be a distributed search peer of the forwarder ? On a pure forwarder (as against a search head that's forwarding it's internal logs ) distsearch.conf shouldn't be listing any search peers.

[emiller42]

Aha, there's a distsearch.conf file on the forwarder for some reason. It's got an old modtime, so assuming it's something leftover from botched automation. I'll get it fixed. Thank you!!

[ashleyherbert]

I haven't tried out the 6.3 forwarder yet, but my best bet would be that it is trying to connect to the cluster master. There is a new feature called 'Indexer Auto-Discovery' which connects to the cluster master to determine which indexers are available. See here: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/indexerdiscovery

Could that be turned on by default?

[emiller42]

It looks like you have to explicitly configure the forwarder to tell it where to go for indexer discovery. Since I haven't set that up, (There is no [indexer_discovery:$foo] stanza on the forwarder) I would be very surprised if it was just trying to reach out to an indexer for that info.