topic Re: How to create and calculate a response time graph? in Splunk Search

How to create and calculate a response time graph?

maniishpawar — Wed, 19 Apr 2017 16:48:24 GMT

How do i calculate every 10 seconds, the average response time for the past 5 minutes and plot on a graph.

Re: How to create and calculate a response time graph?

gokadroid — Wed, 19 Apr 2017 17:52:40 GMT

let's say that the response time is being taken in a field called responseTime and the statement calculate every 10 seconds, the average response time for the past 5 minutes and plot on a graph means what is the average response time calculated for 10 sec spans and when watched over last 5 minutes, here is what you can try:

your query to return field responseTime earliest=-5m
| timechart span=10s avg(responseTime)

Re: How to create and calculate a response time graph?

somesoni2 — Wed, 19 Apr 2017 18:01:22 GMT

So at 10:00:00, show average of 09:55:00 to 10:00:00, at 10:00:10, show average of 09:55:10 to 10:00:10, at 10:00:20, show average of 09:55:20 to 10:00:20, and so on...?

Re: How to create and calculate a response time graph?

maniishpawar — Wed, 19 Apr 2017 18:02:45 GMT

little bit confusing, but to me the answer seems providing average on 10 sec window,
but the avg is required for previous 5 mins. please correct me if I am wrong.

so all in all for 1 hour we will 60*6 =360 samples( each at 10s interval) , each showing me the average of past 5 mins from the collected _timestamp.

Re: How to create and calculate a response time graph?

maniishpawar — Wed, 19 Apr 2017 18:03:50 GMT

Yes correct

Re: How to create and calculate a response time graph?

somesoni2 — Wed, 19 Apr 2017 18:39:41 GMT

What version of Splunk are you using?

Re: How to create and calculate a response time graph?

maniishpawar — Wed, 19 Apr 2017 19:03:43 GMT

we are using splunk cloud

Re: How to create and calculate a response time graph?

woodcock — Wed, 19 Apr 2017 19:07:38 GMT

You create a dashboard that refreshes every 10 seconds (actually, this is going to be WAY too much flashing, so I suggest every 30 seconds at least) and add a single panel to that dashoard that runs a single search and generates a single-value visualization. Then you create the search that drives that panel, something like this:

Your Base Search Here earlieast=-5m@m | stats avg(responseTime) AS AvgResponseTime

Re: How to create and calculate a response time graph?

somesoni2 — Wed, 19 Apr 2017 19:20:22 GMT

Assuming Splunk Cloud is using Splunk version 6.4 or above (check Help-> About), give this a try

your query to return field responseTime 
| bucket span=10s _time
| streamstats time_window=5m avg(responseTime) as "Average Reponse Time"
| dedup _time | table _time  "Average Reponse Time"

Re: How to create and calculate a response time graph?

lguinn2 — Wed, 19 Apr 2017 20:01:11 GMT

If you want to plot a moving average, perhaps you can use the trendline command. I also like @somesoni2's answer too.

 your query to return field 
| sort _time
| trendline sma30(responseTime) AS trend
| table _time responseTime trend

However this answer makes a different assumption about the data. This assumes that you collect responseTime information every 10 seconds. To get a 5-minute average at any point, you would need to average at the prior 30 responseTimes. This is what trendline does.

Re: How to create and calculate a response time graph?

woodcock — Wed, 19 Apr 2017 20:05:27 GMT

Never mind; I missed the "and plot as a graph".

Re: How to create and calculate a response time graph?

DalJeanis — Wed, 19 Apr 2017 21:49:32 GMT

For a smidge more accuracy... and ONLY a smidge...we'd probably do the bucket after the streamstats, and then run it into stats... or just ignore the buckets and run it directly into timechart, like so ...

 your query to return fields _time and responseTime over a 15-minute interval 
earliest=-16m@m 
 | sort 0 _time  
 | streamstats time_window=5m avg(responseTime) as avgResponseTime
 | addinfo
 | bin info_max_time as maxtime span=10s 
 | where (_time >= info_min_time+300) and (_time < maxtime)
 | timechart span=10s avg(avgResponseTime) as avgResponseTime

... It's probably not absolutely necessary to kill the most recent interval, which will never be 10s long, but we did it anyway in that addinfo code. You know, "belt, suspenders, safety pins and duct tape".

Re: How to create and calculate a response time graph?

maniishpawar — Thu, 20 Apr 2017 17:28:51 GMT

I tried this query , its showing the results.
when I click on one timestamp (10s each) , should this be showing me the events of -5m from that timestamp.
As of now, when I clicked lets 4/20/2017 1:20:00 PM timestamp it does not show me the first event being 4/20/2017 1:15:00 PM and last being 4/20/2017 1:20:00 PM

Re: How to create and calculate a response time graph?

somesoni2 — Thu, 20 Apr 2017 20:32:29 GMT

My guess will be no, it won't show you events for 5 min window of the time clicked. It will show the events from time clicked + the timechart span which is 10 sec. For showing results for last 5 min you'll have to setup custom drilldown to take the clicked timestamp and update earliest and latest accordingly.

Re: How to create and calculate a response time graph?

maniishpawar — Mon, 24 Apr 2017 16:06:14 GMT

sorry @somesoni2 , i'm here again. what is the equivalent of streamstats in splunk 6.3.3 ?
or is there a different way to write this in older splunk version

Re: How to create and calculate a response time graph?

DalJeanis — Mon, 24 Apr 2017 16:55:51 GMT

Okay, if you are on splunk below 6.4, then streamstats won't work for you. here's an alternate route. Basically, we copy each record forward into the next twenty-nine 10-second intervals, kill the excess records that go out into the future, and then let timechart do all the work.

  your query to return fields _time and responseTime over a 15-minute interval 
 earliest=-16m@m 
  | bin _time span=10s
  | eval fan_time=mvrange(0,300,10)
  | mvexpand fan_time
  | eval _time=_time+fan_time
  | addinfo
  | bin info_max_time as maxtime span=10s 
  | where (_time >= info_min_time+300) and (_time < maxtime)
  | timechart span=10s avg(responseTime) as avgResponseTime