@wvalente, there could be several ways to identify this.
_internal index has already been called out by Giuseppe. For remaining like:
3) REST api with
| rest /services/deployment/server/clients
Distributed Monitoring Console to monitor deployment status, which uses above REST API.
Refer to answer by @DalJeanis which compiles all these and more 🙂
you can check if your forwarders are sending logs to Splunk, reasons for this situations could be many (splunkd not running, network problems, host down, etc...) but anyway I think that you should check if a forwarder is sending logs and not if slunkd is running, anyway if there are other problems you cannot receive logs from the forwarder!
So, to check if a forwarder is sending logs you can search on _internal
if you don't receive logs there's a problem.
You can create an alert using a lookup (calling e.g. perimeter.csv) containing all the hosts to monitor and running e.g. every 5 minutes the following search:
index=_internal | eval host=upper(host) | stats count by host | append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host count ] | stats sum(count) AS Total by host | where Total=0
In this way, hosts where Total=0 are missing and hosts where Total>0 are sending logs.
You can also show host status in a dashboard (also in graphic mode).