Splunk Search

Alert based on comparison between two results of the same search

JordanPeterson
Path Finder

I have the WinHostMon service reporting in every 5 minutes. I want to be alerted if a State changes from one 5 minute windows to the next. Right now I have a search that looks like this:

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-11m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State _time
| dedup 2 host Name
| sort host Name

Which will get me the last two reported states. Now what I want to do from here is alert if the previous State was "Stopped" and the new State is "Running". How can I compare where each host and name are the same but the state is different? I'd love to return it as a singular result to make alerting easier if possible as well.

If I can get the result like this it would be great:

|host | Name | DisplayName | Description | PreviousState | NewState |

0 Karma
1 Solution

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"

View solution in original post

0 Karma

JordanPeterson
Path Finder

I was able to get what I wanted using the dedup sortby and join commands.

index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-5m 
    [ inputlookup svc.csv] 
| table host Name DisplayName Description State 
| dedup host Name sortby -_time 
| join host Name 
   [ search index="pod-os" sourcetype=WinHostMon source=service StartMode=Auto earliest=-10m 
        [ inputlookup svc.csv] 
    | table host Name DisplayName Description State _time 
    | dedup host Name sortby +_time 
    | rename State as PrevState 
    | fields host Name PrevState ] 
| where PrevState!="Running" AND State="Running"
0 Karma
Get Updates on the Splunk Community!

Observability Unveiled: Navigating OpenTelemetry's Framework and Deployment Options

Observability Unveiled: Navigating OpenTelemetry's Framework and Deployment Options A recent Tech Talk, ...

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...