Splunk Search

How to edit my search to detect a significant change in events frequency over a sliding window?

pm771
Communicator

Based on an event log we would like to find event type which frequency changed by 50% or more over a 5 min window.

Using my limited Splunk knowledge and heavy Googling, I came up with something like this:

index=index_of_events | eval cnt=1 | timechart span=20s limit=40 per_second(cnt) as ev  by ev_typeuseother=f usenull=f |
streamstats window=40 global=false first(ev) as start last(ev) as end by ev_type | 
eval diff=abs(start-end) | eval max_val=max(start, end) | 
where diff > 0 AND max > 0 | eval prc=100*diff/max_val | where prc > 50

I'm getting reasonable output after streamstats, but then I'm losing the data.

Was it OK to pipe timechart directly into streamstats? Did I need untable (or something) in between?

How do I get it right?

0 Karma

sundareshr
Legend

See if this gets you what you're looking for

index=index_of_events | bin span=5m _time | stats count by _time evtype | streamstats window=1 current=f global=f first(count) as start by evtype | eval diff=(abs(start-count)/start)*100 | where diff>50
0 Karma

pm771
Communicator

Unfortunately I do not believe it solves my problem, as it compares adjacent number of events, while I need to compare frequencies. In physics terms, I need to measure drops of speed and not distance. That's why I used per_second().

Have I missed something in your solution?

What is wrong with my query? Can it be somehow salvaged?

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...