Deployment Clients not showing in Monitoring Console?

[testingtena]

Our Splunk instance is currently setup as a deployment server.

All our clients have the Universal Forwarder installed and setup as deployment clients, phoning home to the server to get their necessary apps.

Under the "Forwarder Management" page of the Distributed Environment settings, can see all 20 of our clients and their respective host name and IP address actively talking with the server by phoning home and getting apps deployed...

However, when I go to the Monitoring Console's, "Forwarders: Deployment" page, only 6 of the 20 Universal Forwarders are showing as installed and active?

Sure we're messing up one of the many different config files somewhere but not sure which one...

[PickleRick]

As an expansion of @kprior201 's answer - a bit of an explanation.

Since DS is the component directly responding to queries from the deployment clients, it maintains and displays the list of clients that already "phoned home". But if you restart the DS service, it has to rebuild its database.

On the other hand, MC does not interact directly with the deployment clients in any way. It only monitors the _internal index for logs forwarded from all components in your environment.

So you might have a situation where some forwarders do phone home and get apps from the DS but cannot properly send their events to the indexer layer.

[thahir]

@testingtena  first identify the missing forwarder by using the below query.

index=_internal source="/opt/splunk/var/log/splunk/metrics.log*" sourcetype="splunkd" fwdType="*" | dedup sourceHost | rename IPAddress AS hostip, sourceHost AS IPAddress, OS AS fOS | fields IPAddress, hostname, fGUID, fOS, fwdType.

This will list information about connected forwarders based on logs.

there could be an issue with specific configuration files. Here's what to check:

• deploymentserver.conf on the deployment server: Ensure the configuration allows communication with UFs.
• inputs.conf on the UFs: Verify the stanza forwarding data to the deployment server is correct.

[kprior201]

I would check that the saved search populating the forwarder table in MC is finding the results as expected. Maybe the logs aren't making it from those forwarders that are missing? If you followed the setup instructions below, I don't think you're missing anything glaringly obvious.

Saved search and setup information: https://docs.splunk.com/Documentation/Splunk/9.2.1/DMC/Configureforwardermonitoring