Host state change

[rmorlen]

We have 4 servers running. 2 active and 2 as offline. Doing a search similar to "hostname="MyServers*" sourcetype="status" serverState="*" | stats count by serverState" will give me a list of the servers and their current state (either active or offline). We want to be able to alert when any of the servers change state (go from active to offline or offline to active). Any suggestions on how to do this? I looked at Tracking Hosts Through a State Diagram but it didn't make sense to me.

[chris · Accepted solution]

If the hosts will log their state in regular intervals you could schedule a search that runs once every interval over a span of 2 intervalls and do something like:

basesearch serverState="*" | stats dc(serverState) as states by host | where states>1

This would give you a list of servers that have changed their state if you throw in a latest(serverState) you will also get the state they are in now.

If the events are not logged at regular intervalls you can still detect state changes if you search over big enough time span.
Using a search to update a lookup table or a summary index that keeps trace of the last state and a second search to compare the lookup table/summary index to the current state is probably the way to go (this is what kristian suggests)

[rmorlen]

Yep. This worked. Thanks!

[kristian_kolb]

Perhaps you should have a look at this - using a lookup-table to maintain state (and detect changes)

http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/

/K

[rmorlen]

Whew. Great article. I still can't get when a server changes from active to offline. (so a server was online a minute ago and it is now offline). I can get a list of servers that are either active or offline.