Getting Data In

Changes to forwarder->indexer connectivity in 6.3.0?

emiller42
Motivator

Hello! I've upgraded my test environments to Splunk 6.3, and I'm noticing that my forwarders have begun throwing the following warnings in splunkd.log:

WARN  DistributedPeerManagerHeartbeat - Unable to get server info from peer: http://<IP OF INDEXER>:8089 due to: Connect Timeout; exceeded 10000 milliseconds

Now, I'm not sure why my forwarders are attempting to reach the management port on my indexers in the first place. These environments do not use index clustering, so the only port open between the two is 9997 for forwarding data. Is there some new functionality that needs the indexer management port to be accessible to the forwarder?

I'm not noticing any gaps in indexed data, so there doesn't appear to be any appreciable impact. But the warnings are concerning.

1 Solution

amathew_splunk
Splunk Employee
Splunk Employee

Is the indexer set to be a distributed search peer of the forwarder ? On a pure forwarder (as against a search head that's forwarding it's internal logs ) distsearch.conf shouldn't be listing any search peers.

View solution in original post

amathew_splunk
Splunk Employee
Splunk Employee

Is the indexer set to be a distributed search peer of the forwarder ? On a pure forwarder (as against a search head that's forwarding it's internal logs ) distsearch.conf shouldn't be listing any search peers.

View solution in original post

emiller42
Motivator

Aha, there's a distsearch.conf file on the forwarder for some reason. It's got an old modtime, so assuming it's something leftover from botched automation. I'll get it fixed. Thank you!!

0 Karma

ashleyherbert
Communicator

I haven't tried out the 6.3 forwarder yet, but my best bet would be that it is trying to connect to the cluster master. There is a new feature called 'Indexer Auto-Discovery' which connects to the cluster master to determine which indexers are available. See here: http://docs.splunk.com/Documentation/Splunk/6.3.0/Indexer/indexerdiscovery

Could that be turned on by default?

0 Karma

emiller42
Motivator

It looks like you have to explicitly configure the forwarder to tell it where to go for indexer discovery. Since I haven't set that up, (There is no [indexer_discovery:$foo] stanza on the forwarder) I would be very surprised if it was just trying to reach out to an indexer for that info.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!