Splunk Search

How to calculate processing rate of metrics within specified time range?

hippe21
Explorer

Here's an example snippet of the logs I'm working with:

2018-04-17 18:26:02 app=test-app, env=qa, total_msg=0
2018-04-17 18:25:02 app=test-app, env=qa, total_msg=60
2018-04-17 18:24:02 app=test-app, env=qa, total_msg=0
2018-04-17 18:23:02 app=test-app, env=qa, total_msg=100
2018-04-17 18:22:02 app=test-app, env=qa, total_msg=50

I'd like to create alerts and dashboard around these metrics. I've been attempting to use delta, but it's returning a negative number for my 'msg_proc' value. The query I'm using is:

index=myindex sourcetype=mymetrics environment="qa" app=test-app
| bucket _time span=1m
| stats sum(total_msg) as current by _time, app
| delta current
| rename delta(current) as msg_proc

The above query results in the output below:

_time,app,current,msg_proc
2018-04-17T14:59:00,test-app,0,0
2018-04-17T15:00:00,test-app,0,0
2018-04-17T15:01:00,test-app,42,0
2018-04-17T15:02:00,test-app,27,-15
2018-04-17T15:03:00,test-app,35,8
2018-04-17T15:04:00,test-app,21,-14
2018-04-17T15:05:00,test-app,3,-18
2018-04-17T15:06:00,test-app,1,-2
2018-04-17T15:07:00,test-app,1,0
2018-04-17T15:08:00,test-app,1,0
2018-04-17T15:09:00,test-app,1,0

As expected the delta is showing a negative number (since it's 'processing' that many messages from previous number). I'd like to send an alert when current is >0, and msgs are not being consumed within a 5 minute period (or alert if it's slow). I struggling to define good logic to alert for this condition. I'd like to create a dashboard for specific queues to show the msg_proc rate for each queue if possible. Hope this makes sense!

0 Karma

somesoni2
Revered Legend

Not sure delta would give you anything good. For processing rate, try to use Avg function with stats or timechart.

0 Karma

adonio
Ultra Champion

looks like you are pulling in 60 minutes intervals, maybe try to work with streamstats and its arguments window or time_window
more about the command here:
http://docs.splunk.com/Documentation/Splunk/7.0.3/SearchReference/Streamstats

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...