Splunk Search

% difference in errors over the last 5 minutes

samneo
Path Finder

Im looking to get a query that will tell me the difference in an error rate increase i.e 5 minutes ag it was 120 errors but now is above 10%. My current search is as per the below

 

index=aws_kubernetes app=nio tag=error env=prd* | timechart span=1m count by app limit=0

 

this will show me the standard error rate over time so need to know when a percentage increase happens

Labels (1)
0 Karma
1 Solution

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

View solution in original post

0 Karma

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

0 Karma

samneo
Path Finder

unfortunately this doesnt work, the line

| timechart span=1m sum(count) as count by app limit=0

brings back no results as the timechart has no count and only shows the field nio which has the count in. I used nio in that field instead but still doesnt work as per your one above

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You don't need that line either - use your own timechart line - The whole example is a runanywhere example to show it working - just use the ideas from the bottom part

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| gentimes start=-1 increment=1m
| rename starttime as _time
| eval app=split("ABCD","")
| mvexpand app
| eval count=random()%100




| timechart span=1m sum(count) as count by app limit=0


| untable _time app count
| sort 0 app _time
| streamstats window=1 current=f values(count) as previous by app
| eval increase=if(count>previous,100*count/previous,null())
| xyseries _time app increase
0 Karma

samneo
Path Finder

Hi @ITWhisperer 

 

How do i incorporate that with my index as when i add it on it says 

 

Error in 'gentimes' command: This command must be the first command of a search.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The part before the blank lines just sets up dummy data - you have real data to work with so you don't need this part

0 Karma

samneo
Path Finder

unfortunately this didnt work for my data

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...