Splunk Search

% difference in errors over the last 5 minutes

samneo
Path Finder

Im looking to get a query that will tell me the difference in an error rate increase i.e 5 minutes ag it was 120 errors but now is above 10%. My current search is as per the below

 

index=aws_kubernetes app=nio tag=error env=prd* | timechart span=1m count by app limit=0

 

this will show me the standard error rate over time so need to know when a percentage increase happens

Labels (1)
0 Karma
1 Solution

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

View solution in original post

0 Karma

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

0 Karma

samneo
Path Finder

unfortunately this doesnt work, the line

| timechart span=1m sum(count) as count by app limit=0

brings back no results as the timechart has no count and only shows the field nio which has the count in. I used nio in that field instead but still doesnt work as per your one above

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You don't need that line either - use your own timechart line - The whole example is a runanywhere example to show it working - just use the ideas from the bottom part

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| gentimes start=-1 increment=1m
| rename starttime as _time
| eval app=split("ABCD","")
| mvexpand app
| eval count=random()%100




| timechart span=1m sum(count) as count by app limit=0


| untable _time app count
| sort 0 app _time
| streamstats window=1 current=f values(count) as previous by app
| eval increase=if(count>previous,100*count/previous,null())
| xyseries _time app increase
0 Karma

samneo
Path Finder

Hi @ITWhisperer 

 

How do i incorporate that with my index as when i add it on it says 

 

Error in 'gentimes' command: This command must be the first command of a search.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The part before the blank lines just sets up dummy data - you have real data to work with so you don't need this part

0 Karma

samneo
Path Finder

unfortunately this didnt work for my data

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...