forwarder manager stopped after upgrade from 9.1 to 9.2.0.1

[mykol_j]

Linux, RHEL 8.9. Splunk 9.2.0.1

Had a forwarder manager running (for years) with 2,000+ clients connecting. Did the upgrade from 9.1 to 9.2.0.1 and now have "No clients phoned home."

No firewall or selinux issues are noted.

Getting gazillions of:

03-21-2024 09:59:59.050 -0500 WARN AutoLoadBalancedConnectionStrategy [8459 TcpOutEloop] - Current dest host connection 10.14.8.107:9997, oneTimeClient=0, _events.size()=20, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Thu Mar 21 09:59:45 2024 is using 18446604244100536835 bytes. Total tcpout queue size is 512000. Warningcount=301

Funny thing is, that's the only "error" (warning) I have. it otherwise looks like it's seeing clients:

03-21-2024 09:59:15.468 -0500 INFO PubSubSvr [842449 TcpChannelThread] - Subscribed: channel=tenantService/handshake/reply/carmenw2pc/A265FEF1-4A37-4D58-90ED-AD1142694F05 connectionId=connection_10.14.72.83_8089_blah.domain.edu_blah_A265FEF1-4A37-4D58-90ED-AD1142694F05 listener=0x7f2c78d44000

[mykol_j]

additional:

It appears the forwarder manager is servicing clients, but they are not being reflected in the GUI or at the commandline:

[root@splunkdeployer ~]# /opt/splunk/bin/splunk list deploy-clients

Splunk username: admin
Password:
Login successful, running command...
No deployment clients have contacted this server.

Go figure...

More Googling...

[mykol_j · Accepted solution]

This corrected itself, after I toggled the server's role from standalone to distributed, then back to standalone -- then clients started showing up on the UI.

Monitoring Console, General Setup, Mode (top left).

Go figure.

[RDumbeck]

• this happened to me in a clustered and distributed environment.  I toggled from Distributed to Stand along and back to Distributed then clicked apply changes.  Cleared right up.

[askargbo]

This solved my issue

[isoutamo]

Hi
Have you read this https://docs.splunk.com/Documentation/Splunk/9.2.0/Updating/Upgradepre-9.2deploymentservers ?
r. Ismo

[mykol_j]

No, I haven't, thanks!  Missed this in the release notes...

Will let you know how it works out.

[mykol_j]

Darn. Nope. All those conditions check out OK in my environment. New indexes are where they should be, it's a stand-alone deployment manager, etc..