After upgrade 10.0 to 10.2.0, Forwarder Management Web UI is unavailable

[brycemasterman]

After upgrading, the webpage says Agent management unavailable "there is an error in your serverclass.conf file, which is preventing the deployment server from initializing". I've ran through the spec file a hundred times and there is nothing wrong with the conf file. 

In supervisor.log I noticed these errors:
package start failed splunk-supervisor, packageName agent-manager error: allocate sockets error: error allocating sockets for sidecar agent-manager: IPC broker client error post https://127.0.0.1:8194/v1/registration dial tcp 127.0.0.1:8194: connect: connection refused

The only thing I see that changed was the addition of the app SplunkDeploymentServerConfig. I'm going insane trying to fix this. Please do not reference /etc/hosts. This isn't affecting systems from phoning in and receiving new deployments, but I'd like to resolve the issue.

Current environment:

1 Search Head

2 Indexers (not clustered, but load balanced)

1 Deployment Server

[brycemasterman · Accepted solution]

All,

I just updated with 10.2.1 and it resolved the issues I've had with Agent Management

[kknairr]

@brycemasterman, @partom24   As a workaround, separate Search Head capabilities from the Deployment Server such as Monitoring Console or Cluster Manager to prevent the server from returning API results for other servers (search peers) than itself. 
Ref:
Agent Management page on Deployment Server not loading after Splunk Enterprise 10.2 upgrade | Splunk 

> Marking the answer and giving Karma helps others find solutions faster!

[ktribb]

We attempted this workaround, but unfortunately, it didn't resolve the issue. We're also running RHEL 8, but I noticed someone else is running RHEL 9 in this thread with, presumably, the newer required version of glibc and is reporting the same issue. 

Back to the drawing board. 

[whar_garbl]

I can confirm the problem still exists on RHEL 9. I upgraded one of my clusters after discovering this problem and still have the same behavior. 

[partom24]

Thank you kindly for the response and tagging me!
Sadly, I don't think this helps my specific situation as we are a single server instance. I am speaking to my team to see if it is worth standing up an additional server in the meantime. 

[k_good]

https://splunk.my.site.com/customer/s/article/Agent-Management-page-not-loading-after-Splunk-Enterpr...

[fatsug]

It seems like if your deployment server is moonlighting as something else, the agent management page becomes unavailable. Found a rouge server.conf file setting mode=search and enabling distributed search though search peers.

Removing this file and restarting the service, the agent management page loaded and is fully functional.

So this solved the issue, though I suppose this is only an option if you have a dedicated deployment server and can disable distributed search functionality.

[partom24]

Same issue here. We only have a single server, behind several layers of fire walls and no outside connectivity. 
I've searched all the logs for anything referencing serverclasses or errors and am turning up empty handed.

[firdavs]

Hello have someone found the solution?

[gatundu_]

An upgrade to version 10.2.2 solves the issue

[kduensing]

For us it was 100% due to the lack of support for GLIBC 2.28 which ships with RHEL8. We have a support ticket that verified this. Unfortunately the recommendation is to upgrade the OS to RHEL9, which is not approved yet in my organization.

From the ticket:

"Official Documentation Reference

According to the Splunk 10.2.0 Release Notes and System Requirements:

- Edge Processor/Agent Management: Support for RHEL 8.0 has been removed for these specific components.

- Supported OS: RHEL 9.0+, Ubuntu 24.04 LTS, and Debian 12+ are now the required platforms for the data management control plane and its associated sidecars."

I found documentation for the Edge Processors that state this, but not the Agent Management... so. ¯\_(ツ)_/¯

[firdavs]

I do not think upgrading will help becase in my organization we use the latest RHEL with latest updates

Can someone help me to escalate it to official support of Splunk loosing control of management agents is critical for us!

[whar_garbl]

I don't see anything in the Enterprise docs about it - just Edge Processor (which I don't run). The Enterprise docs for 10.2.0 clearly list RHEL 8 as supported. Ref: https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/10.2/plan-your-splunk-e...

I get the impression that maybe whoever worked on your ticket saw that note about Edge Processor, assumed it was for everything, and jumped on the opportunity to wash their hands of it.

[kduensing]

As far as their reasoning, yes I agree, but If you see my other reply the "splunk-cmp-orchestrator" sidecar is in fact not supported.

[kduensing]

Same issue here.
Air-gapped environment, unable to disable phone home, and agent management unavailable. Will be following closely for a solution.

[kduensing]

Hope this isn't another red herring.
Curious if others are also on RHEL 8 with GLIBC 2.34. The splunk-cmp-orchestrator sidecar which runs when accessing the agent management UI, as evident by tailing the supervisor.log, requires GLIBC 2.34.

Quick check-
ldd /opt/splunk/var/run/supervisor/pkg-run/pkg-cmp-orchestrator*/splunk-cmp-orchestrator

Not prepped for a RHEL9 upgrade, so I hope this is patched soon.

[whar_garbl]

I wonder if you haven't hit on the problem here. RHEL 8 only ships glibc 2.28 for its entire lifecycle. If there truly is a component requiring glibc 2.34 causing this problem, that seems like a huge oversight considering it's supremely impractical to push RHEL to a later glibc within the same major release. There is nary a mention of a glibc version requirement in the 10.2 system requirements, and RHEL 8 is listed as supported, so...

[livehybrid]

Hi @brycemasterman 

It sounds like the sidecar running on port 8194 isnt running.

Can you check for any errors in $SPLUNK_HOME/var/log/sup-pkg-ipc_broker-stdout.log ? 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[brycemasterman]

Thank you for the help.

Checking this specific log I see the following below.

WARN splunk/splunk_client.go message not able retrieve conf stanza value service iopc_broker hostname redacted.com uri https://127.0.0.1:8089/services/properties/server/ipc_broker/agent-manager.agent-manager__control__p... conf file server stanza ipc_broker key agent-manager.agent__control__plane:address error stanza not found

Also, within index=_internal source_type=splunk_assist_internal_log I'm seeing download tests failed to several Splunk websites. I'm assuming post upgrade that splunk_assist wants to update? This won't be possible given our environment. Perhaps I need to download the latest splunk_assist app for this to be corrected?

[ktribb]

Just throwing my hat in the ring. We have the same issue going on and the same type of environment that will not allow us to reach out to the internet.