Archive
Highlighted

indexer discovery mystery - data is flowing but forward-server list is empty

I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:

Active forwards:
None
Configured but inactive forwards:
None

Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIMEWAIT forever until the connection is killed:
tcp 0 0 10.5.3.121:36314 10.10.84.16:8089 TIME
WAIT -

Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.

But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.

The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.

I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.

  • The behaviour is the same whether the forwarder is using 8.0.2.1 or 7.0.13.1
0 Karma
Highlighted

Re: indexer discovery mystery - data is flowing but forward-server list is empty

Never mind, it seems to be something on this particular forwarder. I added a few more, and they're showing up as expected.

0 Karma