How to plot a delta timechart of average response time

[gokadroid]

I have data like:

timestamp, serviceName, responseTime(in ms)

I want to plot the per minute delta of avg. responseTime (difference between avg responseTime yesterday vs today) by serviceName. Average is taken every minute. I want to observe only half an hour window.

Sample data:

03/17/2017 00:00:01 service1 242 
03/17/2017 00:00:02 service2 300
03/17/2017 00:00:03 service3 350 
03/17/2017 00:00:04 service1 280 
03/17/2017 00:00:05 service2 290 
03/17/2017 00:00:06 service3 300 
:
:
03/18/2017 00:00:01 service1 1242 
03/18/2017 00:00:02 service2 1300
03/18/2017 00:00:03 service3 1350 
03/18/2017 00:00:04 service1 1280 
03/18/2017 00:00:05 service2 1290 
03/18/2017 00:00:06 service3 1300 

Now,

- The avg(ResponseTime) of service1 for 03/17/2017 00:00 is (242+280)/2 = 261ms
- The avg(ResponseTime) of service1 for 03/18/2017 00:00 is (1242+1280)/2 = 1261ms
- Hence the delta avg(RespTime) for service 1 at 00:00 between yesterday and today is 1261-261 = 1000ms. It might also be negative 1000 if it was 1261 yesterday and 261 today.

I want to plot this delta by service name on a timechart for a window of last 30 minutes from now only. Please assist.

NOTE

- Services are more than three
- One service might get called mote than other service within a minute. So service1 might get called multiple times within a minute while chances are service2 might not be called at all within that minute.
- There is no sequence in which services are called (sample data makes it look like service1, 2 and 3 are in sequence)

[woodcock · Accepted solution]

Here is a run anywhere example (you will swap your base search and host for service and 1h for 1m😞

index=_introspection sourcetype=splunk_resource_usage 
| timechart span=1h avg(data.reads_kb_ps) AS HourlyAvgResponseTime BY host
| untable _time host HourlyAvgResponseTime
| eval hourmin=strftime(_time, "%H:%M")
| reverse
| streamstats current=f last(HourlyAvgResponseTime) AS prevHourlyAvgResponseTime BY hourmin host
| reverse
| eval delta=HourlyAvgResponseTime-prevHourlyAvgResponseTime

[gcusello]

Hi gokadroid,
see timewrap command (http://docs.splunk.com/Documentation/Splunk/6.5.2/SearchReference/Timewrap)

your_search 
| timechart avg(responsetime) AS responsetime count span=min 
| timewrap 1d align=now 
| sort -_time 
| head 30 
| eval diff=responsetime_latest_day-responsetime_1day_before
| table _time responsetime_latest_day responsetime_1day_before diff
| rename responsetime_latest_day AS Today responsetime_1day_before AS Yesterday diff AS Difference

Using 2 days as time period
Bye.
Giuseppe

[gokadroid]

Thanks for the quick response but can you please provide computing the delta part of it?