Getting Data In

indexer discovery mystery - data is flowing but forward-server list is empty


I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:

Active forwards:
Configured but inactive forwards:

Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIME_WAIT forever until the connection is killed:
tcp 0 0 TIME_WAIT -

Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.

But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.

The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.

I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.

  • The behaviour is the same whether the forwarder is using or
0 Karma


Never mind, it seems to be something on this particular forwarder. I added a few more, and they're showing up as expected.

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...