Environment: - Splunk Enterprise 10.2.7, incrementally upgraded from an older 10.x line - RHEL 8 & Windows Server 2022 Standard - 2 Search Heads (1 PPROD, 1 TEST), 2 Indexer (clustered, not best practice I know. I didn't set this environment up), 1 Cluster Manager/License Manager/Monitoring Console (all-in-one), 1 Heavy Forwarder, 1 Deployment Server Both the Heavy Forwarder and the Deployment Server are running on RHEL 8. Rest of the systems are running on Windows Server 2022 Standard. Symptom: Agent Management page intermittently shows "there is an error in your serverclass.conf file, which is preventing the deployment server from initializing." serverclass.conf is valid; the actual failures are socket allocation errors against the IPC Broker on 127.0.0.1:8194: allocate sockets error: error allocating sockets for sidecar agent-manager: IPC Broker client error: Post "https://127.0.0.1:8194/v1/registration": dial tcp 127.0.0.1:8194: connect: connection refused I've seen a few threads suggesting this is a glibc version mismatch (RHEL 8 shipping 2.28 vs. an alleged 2.34 requirement introduced in 10.2.x). I want to push back on that with what I found, since it seems to be a recurring but unconfirmed theory in several threads: 1. Every relevant binary - agent-manager, ipc_broker, and splunk-cmp-orchestrator - is statically linked: ldd <binary> not a dynamic executable readelf -l <binary> | grep interpreter (empty) A statically linked binary has no runtime dependency on the host's glibc at all. If there were a genuine glibc version requirement, the binary would fail to execute immediately with a loader error, not run cleanly for weeks before an unrelated failure. 2. The actual root cause in our environment: the IPC Broker performs a clean, deliberate self-restart - no panic, no OOM, no segfault in dmesg/journalctl - triggered within tens of milliseconds of its own /v1/health endpoint timing out on a supervisor-initiated probe: 20:54:16.553 supervisor: health check for ipc_broker → context deadline exceeded 20:54:16.585 ipc_broker: "stopping package..." → "Graceful exit after everything shutdown" 20:54:20.101 ipc_broker: restart begins, registers successfully again seconds later Any sidecar (agent-manager, identity, cmp-orchestrator in our logs) that tries to register with the broker during that ~3.5s restart window gets connection refused. Request volume and latency on the broker's registration endpoint look completely normal immediately before and after each of these restarts, so it's not an obvious overload - something is specifically stalling the /v1/health handler, and that handler doesn't appear to log through the same access-log path as /v1/registration, so we can't see what it's blocked on from the application logs alone. These restarts have occurred in irregular bursts going back roughly 3 months in our logs - not on any fixed schedule - which rules out cron/logrotate as a trigger on our end. Questions for anyone who's hit this: - Does anyone else see agent-manager/identity/cmp-orchestrator specifically failing with "connection refused" against 127.0.0.1:8194, correlated with an ipc_broker restart in supervisor.log? - Has anyone captured a pprof or strace trace of ipc_broker during one of these restarts? That's our next step, but if someone's already done it I'd rather not duplicate the work. - Which exact Splunk version(s) are you on? Curious whether this is present across the 10.2.x line or specific to certain patch levels. Separately - and probably a different bug - we're also chronically seeing: IPC Broker request error - expected HTTP 201 but received 422 for sidecar opamp-svc / edge-processor-config only on those two sidecars, going back further than the connection-refused issue. If anyone's seen that 422 pattern specifically, curious whether it's related or a separate registration validation issue. I will continue on troubleshooting this and hopefully get another time where the agent management goes offline. The splunkd process was healthy when I first observed it but for the sake of production, I just restarted the Deployment-Server or rather the agent management by running "reload deploy-server", which made the agent management come back up again. Whenever I find something new during my troubleshooting, I will update this post. Thanks in advance to anyone willing to help and suggesting things! 🙂 Happy to share more log snippets if useful.
... View more
