I'm standing up a 7.3.3 index cluster and I have a strange mystery.
I've got the cluster master and search-heads happily forwarding away to the index cluster, and it shows exactly that in the list forward-server output.
I'm starting to set up endpoints*, and I'm using the EXACT same outputs.conf and certs as I'm using on the master and search-heads, and data forwards happily and shows up in searches, but list forward-server shows:
Active forwards:
None
Configured but inactive forwards:
None
Netstat on the forwarder shows that the request goes out to the master over :8089 as configured, but it is never answered, so it just sits at TIME_WAIT forever until the connection is killed:
tcp 0 0 10.5.3.121:36314 10.10.84.16:8089 TIME_WAIT -
Netstat on the master shows the connection request, but it also just says TIME_WAIT until it is killed.
But clearly the forwarder is picking up the indexer discovery data somewhere, because it is forwarding to all 6 members of my cluster in rotation. I know it's not keeping a previous list like it would if the master went down, because it is a fresh install.
The only in the logs on the forwarder except connections to the indexers and notes about log files is:
04-01-2020 11:04:22.632 -0400 INFO TcpOutputProc - Initialization time for indexer discovery service for default group=splunkssl has been completed.
The master doesn't mention this forwarder at all in splunkd.log.
I know that the forward-server list is a bit unnecessary, as the data is being ingested as it should be, but something is not right.
- The behaviour is the same whether the forwarder is using 8.0.2.1 or 7.0.13.1
