Splunk Search

how to compare avg server response time with the one from the current day?

zooky92
New Member

Hello,

here is what I try to do: I want to know if my todays average duration is slower or faster than my average duration from the last 7 days. If it is higher I want to receive a notification. Assuming that right now we have 2pm I want todays average from 00am until 2pm compared to the average of the same timespan but for the last 7 days. I came up with a solution in 2 separate graphs which looks like this:

1: todays average

index=myindex host=myhost
source="*access_log" duration!="" NOT "status" (date_hour > 00 AND date_hour < now)|eval apacheDuration=apacheDuration/1000 |stats avg(duration) by host

2: avarage of the last 7 days for the same timespan

index=myindex* host="my host
source="*access_log" duration!="" NOT "status" earliest=-7d latest=now (date_hour > 00 AND date_hour < now) | stats avg(duration) by host |eval apacheDuration=apacheDuration/1000

Is there a way to combine those to in 1 graph and make it able to send me notifications if the average time of today is higher than the average? It would be even better if I could plot the deviation.

I thought about something like this which obviously doesn't work:

  1. dataset:

    index=myindex host=myhost
    source="*access_log" duration!="" NOT "status" (date_hour > 00 AND date_hour < now)|eval apacheDuration=apacheDuration/1000 |stats avg(duration) by host as avaragethoday

  2. dataset

    index=myindex* host="my host
    source="*access_log" duration!="" NOT "status" earliest=-7d latest=now (date_hour > 00 AND date_hour < now) | stats avg(duration) by host as avarageWeek |eval apacheDuration=apacheDuration/1000

eval spike=if(avarageToday >1.2 * avarageWeek, avarageToday, 0) + show em the difference between avarageToday and avarageWeek

Tags (2)
0 Karma

soumyasaha25
Contributor

Hi @zooky92 try something like the below query.

index=myindex host=myhost source="*access_log" duration!="" NOT "status" earliest=-24h latest=-1h (date_hour > 00 AND date_hour < 14)
|eval apacheDuration_today=apacheDuration/1000 
|stats avg(apacheDuration_today) as avg_apacheDuration_today by host
| appendcols
    [search index=myindex host=myhost source="*access_log" duration!="" NOT "status" earliest=-8d latest=-1d (date_hour > 00 AND date_hour < 14) 
    |eval apacheDuration_week=apacheDuration/1000
    | stats avg(apacheDuration_week) avg_apacheDuration_week by host]
| eval is_spike=if(avg_apacheDuration_today >1.2 * avg_apacheDuration_week, avarageToday, 0)
| stats values(spike_value) as spike_value values(apacheDuration_today) as apacheDuration_today values(avg_apacheDuration_week) as avg_apacheDuration_week values(is_spike) by _time

you can try to run it as a scheduled search at 15:00.

0 Karma

zooky92
New Member

anyone? :S

0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...