Splunk Search

% difference in errors over the last 5 minutes

samneo
Path Finder

Im looking to get a query that will tell me the difference in an error rate increase i.e 5 minutes ag it was 120 errors but now is above 10%. My current search is as per the below

 

index=aws_kubernetes app=nio tag=error env=prd* | timechart span=1m count by app limit=0

 

this will show me the standard error rate over time so need to know when a percentage increase happens

Labels (1)
0 Karma
1 Solution

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

View solution in original post

0 Karma

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

0 Karma

samneo
Path Finder

unfortunately this doesnt work, the line

| timechart span=1m sum(count) as count by app limit=0

brings back no results as the timechart has no count and only shows the field nio which has the count in. I used nio in that field instead but still doesnt work as per your one above

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You don't need that line either - use your own timechart line - The whole example is a runanywhere example to show it working - just use the ideas from the bottom part

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| gentimes start=-1 increment=1m
| rename starttime as _time
| eval app=split("ABCD","")
| mvexpand app
| eval count=random()%100




| timechart span=1m sum(count) as count by app limit=0


| untable _time app count
| sort 0 app _time
| streamstats window=1 current=f values(count) as previous by app
| eval increase=if(count>previous,100*count/previous,null())
| xyseries _time app increase
0 Karma

samneo
Path Finder

Hi @ITWhisperer 

 

How do i incorporate that with my index as when i add it on it says 

 

Error in 'gentimes' command: This command must be the first command of a search.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The part before the blank lines just sets up dummy data - you have real data to work with so you don't need this part

0 Karma

samneo
Path Finder

unfortunately this didnt work for my data

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...