Splunk Search

How to calculate downtime based on the amount of requests an application server processes?

Path Finder

Hi guys. I want to be able to calculate downtime based on the amount of requests that an Application server processes. The downtime is calculated based on the following rules.

  1. Choose a time-span 30 min before and 30 min after the actual downtime.
  2. Calculate the average amount of events based on the top 20 results i.e the 20 minutes with the most amount of processed requests.
  3. Cassify all events as downtime that has 80% or below of the average described in step 2 above.

Below is an example of the result I want to calculate downtime on:

alt text

Tags (2)
0 Karma

Splunk Employee
Splunk Employee

Here is my method to get the top 80% count, using the percentile top 80% counts, and qualify every minute as up or downtime based on this value.

index=_internal source=*web* req_time =*
|  bucket _time span=1m | stats count by _time
| eventstats perc80(count) AS maxperc80 
| eval status=if(count < maxperc80, "down", "up")

You probably want to add some sort of count of consecutive durations and exclude the outliers
Then do the sum of the "down" minutes.

| stats count by status
0 Karma

New Member

...|top 20 status| stats avg(count)

0 Karma

Path Finder

hi, one more things. how do we add step number 2 above to the search where we take the average of the top 20 results.

0 Karma

Esteemed Legend

I know this is not what you are asking but, based on your example which shows an obvious 100% (full vs. partial) outage, why would you not use something like this:

... | streamstats current=f  latest(_time) AS prevEventTime latest(_raw) AS prevEvent | eval downtime = _time - _prevEventTime | where downtime > 100
0 Karma

Path Finder

Thanks for your input. I have something similar in-place already, however point number 2 above is an important part of the search to be able to calculate the downtime in a proper way.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...