Splunk Search

How can I compare 2 latest events and search for the second latest event by host?

packland
Path Finder

Hi,

I'd like to create a search that detects a failover, i.e. it would compare the two latest events by host and when the second last is true, but the last is false it would appear in the results.

For example if I have data coming from 5 hosts, each of which has a single true or false value sent every 2 - 3 minutes, in my case indicating connectivity, I'd like to know pretty much as soon as the first false message comes through. The closest I've come is using the "latest" function to find the latest event and then maybe using a subsearch to find the second latest and compare the two, however as far as I can find there isn't a way to search for the second latest event (like a "latest(up) - 1" kind of thing).

I'm open to doing it in a different way if this isn't the right way to go about it. I know I could use transactions and search for a "false" transaction that is only 1 - 2 events long, however I feel like that's an expensive way of accomplishing something that might be much simpler.

Any suggestions would be great, thanks.

0 Karma
1 Solution

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

View solution in original post

micahkemp
Champion

One potential option is to dedup by host, and keep 2 events per host. These should be the most recent event. Once you have those, you can use earliest/latest against the state, and then search to find the hosts that changed from active to standby:

| makeresults
| append [| makeresults | eval _time=1002, host="A", state="standby"]
| append [| makeresults | eval _time=1001, host="B", state="active"]
| append [| makeresults | eval _time=1000, host="A", state="active"]
| append [| makeresults | eval _time=999, host="B", state="active"]
| append [| makeresults | eval _time=998, host="A", state="standby"]
| append [| makeresults | eval _time=997, host="A", state="active"]
| append [| makeresults | eval _time=996, host="A", state="active"]
| append [| makeresults | eval _time=995, host="B", state="standby"]
| append [| makeresults | eval _time=994, host="B", state="standby"]
| dedup 2 host
| stats earliest(state) AS earliest_state, latest(state) AS latest_state BY host
| search earliest_state=active latest_state=standby

packland
Path Finder

Did not realise you could use dedup like that. Thanks!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...