We have hit-and-miss identification of servers that fall off of Splunk monitoring. There needs to be a critical alert if a non-decommissioned server:
1.Stops reporting to Splunk, or
2.Stops phoning home to the deployment server
Is there a weay to query the rest api from the search head to determind Deployment server contact?
Any help is much apprreciated..
If your deployment server is forwarding its internal logs to your indexing layer, you should be able to use a query like:
index=_internal host="Your_deployment_server_hostname" "/services/broker/phonehome/"
| stats max(_time) AS last_checkin_epoch by clientip
| eval now_epoch=now()
| eval time_since_last_checkin=now_epoch-last_checkin_epoch
| sort - time_since_last_checkin