Splunk Search

How can I compare 2 latest events and search for the second latest event by host?

packland
Path Finder

Hi,

I'd like to create a search that detects a failover, i.e. it would compare the two latest events by host and when the second last is true, but the last is false it would appear in the results.

For example if I have data coming from 5 hosts, each of which has a single true or false value sent every 2 - 3 minutes, in my case indicating connectivity, I'd like to know pretty much as soon as the first false message comes through. The closest I've come is using the "latest" function to find the latest event and then maybe using a subsearch to find the second latest and compare the two, however as far as I can find there isn't a way to search for the second latest event (like a "latest(up) - 1" kind of thing).

I'm open to doing it in a different way if this isn't the right way to go about it. I know I could use transactions and search for a "false" transaction that is only 1 - 2 events long, however I feel like that's an expensive way of accomplishing something that might be much simpler.

Any suggestions would be great, thanks.

0 Karma
1 Solution

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

View solution in original post

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

packland
Path Finder

Did not realise you could use dedup like that. Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...