I am trying to create a dashboard search to monitor if the named process is running on our name servers. I am trying to run a search on a set number of hosts, but if the hosts contain no data set the graph to red.
Here is my search:
index="os" source="top" host=ns1 OR host=ns2 OR host=ns3 OR host=ns4 named earliest=-2min latest=-1min | stats count by host | eval redCount = if(count !=1,count,0) | eval greenCount = if(count = 1,count,0) |fields - count
So basically, I'm looking in the last few minutes that the named process is showing up in the search. Then checking that there is only 1 line per host. If there is only 1 line the graph is green, if the number is not equal to one, turn red.
My issue is, what if I have a host like ns4 that is down and shows no data from the search? Then the search just shows 3 out of the 4 hosts as green.
How do I generate an error if any of the 4 hosts show no data? I am only interested in the 4 "hardcoded" hosts in my search.
Depends what you mean by "generate an error"
The best way would be to have the following search saved as an alert:
index="os" source="top" host=ns1 OR host=ns2 OR host=ns3 OR host=ns4 named earliest=-2min latest=-1min | stats count by host | search count=0
Then set the trigger condition for the alert to be more results than zero. That way you don't actually have to look at the graph, Splunk will tell you if something is wrong.
For more information see the alerting manual: http://docs.splunk.com/Documentation/Splunk/6.4.0/Alert/Aboutalerts
Well I am not sure that works. If i substitute an invalid host lets say ns5 that does not exist. I would expect to see I was trying to create a bar graph with 4 bars for each host. The issue I trying to solve was to keep 4 bar graphs when data is only received from 3 hosts. Then it will show 3 green bar graphs for the process. Then it would have an area for the host that shows no data.
I am trying to create a dashboard for the helpdesk to be aware of when the DNS/named process is not running on the dns servers. Then give them some window into how to monitor this via something visual. I tried to create a single search for 4 servers using bar graphs. If they are used to seeing 4 green bar graphs then if they only see 3 of the 4. Then they know they need to escalate this.
My initial search works. It just doesnt work if there is no data. I want to ensure if there no data from one of the hosts it shows as a color or something visual they can see. I want to maintain the other 3 green graphs for the 3 working servers. I just dont know how to keep those 4 "hardcoded" hosts in the search even if one of them has no data. Then create a visual red/failed item for the users to realize there is an action needed to be taken.