Splunk Search

How to edit my search to detect a significant change in events frequency over a sliding window?

pm771
Communicator

Based on an event log we would like to find event type which frequency changed by 50% or more over a 5 min window.

Using my limited Splunk knowledge and heavy Googling, I came up with something like this:

index=index_of_events | eval cnt=1 | timechart span=20s limit=40 per_second(cnt) as ev  by ev_typeuseother=f usenull=f |
streamstats window=40 global=false first(ev) as start last(ev) as end by ev_type | 
eval diff=abs(start-end) | eval max_val=max(start, end) | 
where diff > 0 AND max > 0 | eval prc=100*diff/max_val | where prc > 50

I'm getting reasonable output after streamstats, but then I'm losing the data.

Was it OK to pipe timechart directly into streamstats? Did I need untable (or something) in between?

How do I get it right?

0 Karma

sundareshr
Legend

See if this gets you what you're looking for

index=index_of_events | bin span=5m _time | stats count by _time evtype | streamstats window=1 current=f global=f first(count) as start by evtype | eval diff=(abs(start-count)/start)*100 | where diff>50
0 Karma

pm771
Communicator

Unfortunately I do not believe it solves my problem, as it compares adjacent number of events, while I need to compare frequencies. In physics terms, I need to measure drops of speed and not distance. That's why I used per_second().

Have I missed something in your solution?

What is wrong with my query? Can it be somehow salvaged?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...