Splunk Search

How to check if a value in a metric continues to grow?

llappall
Observer

I have a metric from AWS for the number of messages visible in a SQS queue, which gets computed every 5 minutes. 

2023-08-02 11:50:00    13.3
2023-08-02 11:55:00    0.0
2023-08-02 12:00:00    33.8
2023-08-02 12:05:00    0.0

This means that there were 13 messages in the queue, and 5 minutes later they were gone (processed). Then there were 33, then they were gone (processed)

If messages do not get processed, I'd expect this number to continue to grow and not decrease. 

I need to set up an alert when that happens. Is there some way to alert when a value grows, say, over 5 rows? 

Or, is there a way to compare a value to itself at different points in time? 

 

Labels (2)
Tags (1)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Basically, comparisons are done between fields in the same "event" - for your use case, this means you have to arrange for values from other events to be available in your event.

There are a number of ways to do this, and it depends on what you are looking to compare.

For example, you could find the change between two consecutive events

| streamstats window=2 current=f latest(sqs) as previous_sqs
| eval sqs_delta=sqs - previous_sqs

Or you could evaluate this as an increase or not, then count how many times it increased over the last 5 events

| streamstats window=2 current=f latest(sqs) as previous_sqs
| eval increase = if (previous_sqs < sqs, 1, 0)
| streamstats window=5 sum(increase) as rolling_increase

Then you could set your alert based on the value of rolling_increase.

Get Updates on the Splunk Community!

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...

Want to Reduce Costs, Mitigate Risk, Improve Performance, or Increase Efficiencies? ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...