Show status of AD services as "Down", when Splunk forwarder not running or system is down ??

[pgadhari]

I am showing the Status of various AD services in a dashboard in the form of icons. Green check icon means service is running, and when the particular service is down, it showns red x-circle icon. But now, the problem is - when the Splunk forwarder is down on the server or the server itself is down, it does not show that server in the table, as the events are not getting into the Splunk.

I want to show "x-circle" icon (critical icon), on all the services, when the server is down or Splunk forwarder is not running which should visualize as server is down or forwarder is down. As of now, it is removing that server from the table list, due to which the support team does not know whether this server was there or not. I am attaching a screenshot for the same, and also adding the query for reference. I think it should be like - when the AD server is down or forwarder on that server is down and events are not coming into Splunk, then Splunk should recogize this and show all services are down, instead of removing server from the list, as the search is not resulting that server events. Please help to resolve this.

index=windows | search Name=NTDS OR Name=ADWS OR Name=DFSR OR Name=DNS OR  Name=kdc OR Name=Netlogon OR Name=Ismserv OR Name=gpsvc OR Name=w32time OR Name=SplunkForwarder | eval Name=upper(Name) |stats values(State) by host,Name | rename host as Server values(State) as Status | eval Status=if(Status="Running","True","False") | xyseries Server Name Status | table Server ADWS DFSR DNS GPSVC ISMSERV KDC NTDS W32TIME NETLOGON SPLUNKFORWARDER

[niketn]

@pgadhari you can adopt one of the two approaches

1) If you maintain inventory of all servers in lookup/kvstore or in DB, you can use SPL to show missing servers as Red cross icon. Refer to the following answer: https://answers.splunk.com/answers/614029/how-to-alert-when-we-are-receiving-data-from-hosts.html

2) Use REST API to find all deployment clients which are configured. If data collection is every 5 minutes lets say, then any Deployment Client which has not phonedHome (based on the lastPhoneHomeTime returned by REST API) in last 5 minute is also down. https://answers.splunk.com/answers/554469/if-my-time-last-connected-time-is-more-than-2-days.html

Following is the link to REST API for deployment client: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTdeploy#deployment.2Fserver.2Fclients

[pgadhari]

I will check this out and revert. Thanks.