Getting Data In

indexer discovery mystery - data is flowing but forward-server list is empty


I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:

Active forwards:
Configured but inactive forwards:

Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIME_WAIT forever until the connection is killed:
tcp 0 0 TIME_WAIT -

Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.

But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.

The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.

I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.

  • The behaviour is the same whether the forwarder is using or
0 Karma


Never mind, it seems to be something on this particular forwarder. I added a few more, and they're showing up as expected.

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!


Or Learn More in Our Blog >>