Splunk Search

How can I compare 2 latest events and search for the second latest event by host?

packland
Path Finder

Hi,

I'd like to create a search that detects a failover, i.e. it would compare the two latest events by host and when the second last is true, but the last is false it would appear in the results.

For example if I have data coming from 5 hosts, each of which has a single true or false value sent every 2 - 3 minutes, in my case indicating connectivity, I'd like to know pretty much as soon as the first false message comes through. The closest I've come is using the "latest" function to find the latest event and then maybe using a subsearch to find the second latest and compare the two, however as far as I can find there isn't a way to search for the second latest event (like a "latest(up) - 1" kind of thing).

I'm open to doing it in a different way if this isn't the right way to go about it. I know I could use transactions and search for a "false" transaction that is only 1 - 2 events long, however I feel like that's an expensive way of accomplishing something that might be much simpler.

Any suggestions would be great, thanks.

0 Karma
1 Solution

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

View solution in original post

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

packland
Path Finder

Did not realise you could use dedup like that. Thanks!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...