Getting Data In

indexer discovery mystery - data is flowing but forward-server list is empty

duke_splunk_adm
Engager

I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:

Active forwards:
None
Configured but inactive forwards:
None

Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIME_WAIT forever until the connection is killed:
tcp 0 0 10.5.3.121:36314 10.10.84.16:8089 TIME_WAIT -

Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.

But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.

The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.

I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.

  • The behaviour is the same whether the forwarder is using 8.0.2.1 or 7.0.13.1
0 Karma

duke_splunk_adm
Engager

Never mind, it seems to be something on this particular forwarder. I added a few more, and they're showing up as expected.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...