Getting Data In

forwarder manager stopped after upgrade from 9.1 to 9.2.0.1

mykol_j
Communicator
Linux, RHEL 8.9. Splunk 9.2.0.1
 
Had a forwarder manager running (for years) with 2,000+ clients connecting. Did the upgrade from 9.1 to 9.2.0.1 and now have "No clients phoned home."
 
No firewall or selinux issues are noted.
 
Getting gazillions of:
03-21-2024 09:59:59.050 -0500 WARN AutoLoadBalancedConnectionStrategy [8459 TcpOutEloop] - Current dest host connection 10.14.8.107:9997, oneTimeClient=0, _events.size()=20, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Thu Mar 21 09:59:45 2024 is using 18446604244100536835 bytes. Total tcpout queue size is 512000. Warningcount=301
 
Funny thing is, that's the only "error" (warning) I have. it otherwise looks like it's seeing clients:
 
03-21-2024 09:59:15.468 -0500 INFO PubSubSvr [842449 TcpChannelThread] - Subscribed: channel=tenantService/handshake/reply/carmenw2pc/A265FEF1-4A37-4D58-90ED-AD1142694F05 connectionId=connection_10.14.72.83_8089_blah.domain.edu_blah_A265FEF1-4A37-4D58-90ED-AD1142694F05 listener=0x7f2c78d44000
Labels (1)
0 Karma
1 Solution

mykol_j
Communicator

This corrected itself, after I toggled the server's role from standalone to distributed, then back to standalone -- then clients started showing up on the UI.

Monitoring Console, General Setup, Mode (top left).

Go figure.

View solution in original post

mykol_j
Communicator

additional:

It appears the forwarder manager is servicing clients, but they are not being reflected in the GUI or at the commandline:

[root@splunkdeployer ~]# /opt/splunk/bin/splunk list deploy-clients


Splunk username: admin
Password:
Login successful, running command...
No deployment clients have contacted this server.

Go figure...

More Googling...

0 Karma

mykol_j
Communicator

This corrected itself, after I toggled the server's role from standalone to distributed, then back to standalone -- then clients started showing up on the UI.

Monitoring Console, General Setup, Mode (top left).

Go figure.

askargbo
Engager

This solved my issue

0 Karma

isoutamo
SplunkTrust
SplunkTrust
0 Karma

mykol_j
Communicator

No, I haven't, thanks!  Missed this in the release notes...

Will let you know how it works out.

0 Karma

mykol_j
Communicator

Darn. Nope. All those conditions check out OK in my environment. New indexes are where they should be, it's a stand-alone deployment manager, etc..

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...