Hi Team, I'm investigating a Forwarder Management count mismatch issue on two Deployment Servers. Environment Splunk Enterprise 9.4.12 Deployment Server #1: ~30,000 clients Deployment Server #2: ~5,000 clients Observation From the Deployment Server REST endpoint: | rest splunk_server=local count=0 /services/deployment/server/clients | stats dc(guid) as GUIDs values(guid) as guid by hostname | sort - GUIDs We observed: Deployment Server #1 10,000 hostnames in exported results 17,176 unique GUIDs 1,086 hostnames associated with more than one GUID Maximum GUIDs for a single hostname: 26 Deployment Server #2 10,000 hostnames in exported results 14,779 unique GUIDs 964 hostnames associated with more than one GUID Maximum GUIDs for a single hostname: 21 No duplicate GUIDs were identified. Instead, the same hostname appears to be associated with multiple unique GUIDs. Question Can multiple GUID registrations for the same hostname cause the Forwarder Management page to report a higher than expected forwarder count? Specifically: Does Forwarder Management count Deployment Clients based on GUID records rather than unique hostnames? Are historical/stale GUID registrations retained in /services/deployment/server/clients and included in the displayed count? Is there a documented or known behavior where rebuilt/reimaged forwarders generate new GUIDs and inflate Forwarder Management client counts? Is there a supported method to identify and remove stale Deployment Client records associated with old GUIDs? Additional Validation We're planning to compare: | rest splunk_server=local count=0 /services/deployment/server/clients | stats count as ClientRecords dc(guid) as UniqueGUIDs dc(hostname) as UniqueHostnames against the count displayed in Forwarder Management. Has anyone observed a similar mismatch where the Forwarder Management count aligns with GUID records rather than unique hostnames? Any guidance or references would be appreciated. Thanks.
... View more