I am trying to build a query through which we can track if all the Splunk forwarders are connected to Cluster Master. Wanted to create an alert if there are issues when forwarder is not able to connect with Cluster master.
Could you please help with the query.
The cluster master already provides the functionality that you're looking for.
From the web UI on the master, go to Settings > Monitoring Console > Forwarders > Forwarders:Deployment
This dashboards provides all the information you're looking for on your forwarders. Status, last connection time, thruput, and so on.
There's also a pre-built alert available for "Missing Forwarders". You simply have to enable it:
Settings > Searches, Reports, and Alerts > App: Monitoring Console > DMC Alert - Missing Forwarders
you could check if an Universal Forwarder is sending internal logs to Indexers and this means that the connection is ok.
To do this, you have to create a lookup with the server to monitor list (called e.g. perimeter.csv) and run something like this:
| metasearch index=_internal | eval host=lower(host) | stats count BY host | append [ | inputlookup perimeter.csv | eval host=lower(host), count=0 | fields host count ] | stats sum(count) AS total BY host | where host=0
Thanks for the response. Exact requirement is to track the forwarders which are not connected to Cluster master, which means that indexer discovery is set in outputs.conf of that forwarder and it is directly sending logs through IP address and not through the cluster master url
I'm a little confused by your last reply. You have indexer discovery configured on the forwarder(s) in question, but they are not communicating with the master? I may be misunderstanding your scenario, but with indexer discovery configured, forwarders will always communicate with the master to get a list of available indexers.
sorry I wasn't so clear!
My hint is to to deploy (using Deployment Server) to all your Universal Forwarders a TA with outputs.conf.
In this way you can be sure about the configuration of your outputs.conf in all UFs.
Then in your outputs.conf you'll surely configure indexer discovery (it's a best practice), but the issue is that in this way you can have a better control on your outputs.conf and be sure that all the UFs are correctly configured.
sorry for the misunderstanding!
did you manage outputs.conf manually or by Deployment Server?
It's a good practice to create a Technical Add-On (called e.g. TA_Forwarders) containing only two files:
and deploy it in all your UFs.
In this way you can centrally manage your UFs and you haven't the problem.