Splunk Search

% difference in errors over the last 5 minutes

samneo
Path Finder

Im looking to get a query that will tell me the difference in an error rate increase i.e 5 minutes ag it was 120 errors but now is above 10%. My current search is as per the below

 

index=aws_kubernetes app=nio tag=error env=prd* | timechart span=1m count by app limit=0

 

this will show me the standard error rate over time so need to know when a percentage increase happens

Labels (1)
0 Karma
1 Solution

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

View solution in original post

0 Karma

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

0 Karma

samneo
Path Finder

unfortunately this doesnt work, the line

| timechart span=1m sum(count) as count by app limit=0

brings back no results as the timechart has no count and only shows the field nio which has the count in. I used nio in that field instead but still doesnt work as per your one above

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You don't need that line either - use your own timechart line - The whole example is a runanywhere example to show it working - just use the ideas from the bottom part

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| gentimes start=-1 increment=1m
| rename starttime as _time
| eval app=split("ABCD","")
| mvexpand app
| eval count=random()%100




| timechart span=1m sum(count) as count by app limit=0


| untable _time app count
| sort 0 app _time
| streamstats window=1 current=f values(count) as previous by app
| eval increase=if(count>previous,100*count/previous,null())
| xyseries _time app increase
0 Karma

samneo
Path Finder

Hi @ITWhisperer 

 

How do i incorporate that with my index as when i add it on it says 

 

Error in 'gentimes' command: This command must be the first command of a search.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The part before the blank lines just sets up dummy data - you have real data to work with so you don't need this part

0 Karma

samneo
Path Finder

unfortunately this didnt work for my data

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...