indexer discovery mystery - data is flowing but forward-server list is empty

I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:

Active forwards:
Configured but inactive forwards:

Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIME_WAIT forever until the connection is killed:
tcp 0 0 TIME_WAIT -

Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.

But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.

The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.

I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.

  • The behaviour is the same whether the forwarder is using or
0 Karma

Never mind, it seems to be something on this particular forwarder. I added a few more, and they're showing up as expected.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!