Splunk Search

How to calculate downtime based on the amount of requests an application server processes?

Norling80
Path Finder

Hi guys. I want to be able to calculate downtime based on the amount of requests that an Application server processes. The downtime is calculated based on the following rules.

  1. Choose a time-span 30 min before and 30 min after the actual downtime.
  2. Calculate the average amount of events based on the top 20 results i.e the 20 minutes with the most amount of processed requests.
  3. Cassify all events as downtime that has 80% or below of the average described in step 2 above.

Below is an example of the result I want to calculate downtime on:

alt text

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

Here is my method to get the top 80% count, using the percentile top 80% counts, and qualify every minute as up or downtime based on this value.

index=_internal source=*web* req_time =*
|  bucket _time span=1m | stats count by _time
| eventstats perc80(count) AS maxperc80 
| eval status=if(count < maxperc80, "down", "up")

You probably want to add some sort of count of consecutive durations and exclude the outliers
Then do the sum of the "down" minutes.

| stats count by status
0 Karma

Archana21
New Member

...|top 20 status| stats avg(count)

0 Karma

Norling80
Path Finder

hi, one more things. how do we add step number 2 above to the search where we take the average of the top 20 results.

0 Karma

woodcock
Esteemed Legend

I know this is not what you are asking but, based on your example which shows an obvious 100% (full vs. partial) outage, why would you not use something like this:

... | streamstats current=f  latest(_time) AS prevEventTime latest(_raw) AS prevEvent | eval downtime = _time - _prevEventTime | where downtime > 100
0 Karma

Norling80
Path Finder

Thanks for your input. I have something similar in-place already, however point number 2 above is an important part of the search to be able to calculate the downtime in a proper way.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.