Splunk Search

% difference in errors over the last 5 minutes

samneo
Path Finder

Im looking to get a query that will tell me the difference in an error rate increase i.e 5 minutes ag it was 120 errors but now is above 10%. My current search is as per the below

 

index=aws_kubernetes app=nio tag=error env=prd* | timechart span=1m count by app limit=0

 

this will show me the standard error rate over time so need to know when a percentage increase happens

Labels (1)
0 Karma
1 Solution

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

View solution in original post

0 Karma

samneo
Path Finder

i ended up using the below with anomaly detection which got me as close in the timeframe i had

 

index=kubernetes app=nio env=prd tag=error
| timechart span=1m count by app limit=0
| eventstats median("nio") as median
| eval absDev=(abs('nio'-median))
| eventstats median(absDev) as medianAbsDev
| eval lowerBound=(median-medianAbsDev*exact(4)), upperBound=(median+medianAbsDev*exact(3))
| eval isOutlier=if('nio' < lowerBound OR 'nio' > upperBound, 1, 0)
| fields _time, "nio", lowerBound, upperBound, isOutlier, *

0 Karma

samneo
Path Finder

unfortunately this doesnt work, the line

| timechart span=1m sum(count) as count by app limit=0

brings back no results as the timechart has no count and only shows the field nio which has the count in. I used nio in that field instead but still doesnt work as per your one above

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You don't need that line either - use your own timechart line - The whole example is a runanywhere example to show it working - just use the ideas from the bottom part

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| gentimes start=-1 increment=1m
| rename starttime as _time
| eval app=split("ABCD","")
| mvexpand app
| eval count=random()%100




| timechart span=1m sum(count) as count by app limit=0


| untable _time app count
| sort 0 app _time
| streamstats window=1 current=f values(count) as previous by app
| eval increase=if(count>previous,100*count/previous,null())
| xyseries _time app increase
0 Karma

samneo
Path Finder

Hi @ITWhisperer 

 

How do i incorporate that with my index as when i add it on it says 

 

Error in 'gentimes' command: This command must be the first command of a search.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The part before the blank lines just sets up dummy data - you have real data to work with so you don't need this part

0 Karma

samneo
Path Finder

unfortunately this didnt work for my data

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...