Plot up or down state over time?

[Jason]

I have a dataset I just created using transaction that shows when a particular service is down by pulling in the "service down" message and the "service up" message.

How do I plot when the service is up vs down on a graph over time?

Similar to the question here.

[Jason]

It just so happens the important service is my splunk indexing, and I'm tracking when the queues become so full the network ports are shut off.

I found a way to do this, but it requires a set time range, because you have to hack _time a few times in the search. The idea is to turn the single "on" events into four events - the beginning and end times of the "on" status - and the "off" status just before and after the "on". Then use "connect points" to create the shape!

The idea is this:

1. Create the transactions, or pull out some sort of events that you can create a starttime and endtime from.
2. In this case, I used periods when the indexer was unable to receive data because its network port was closed due to over 5 minutes of blocked queues. Transaction gives me _time and _time + duration to use.
3. Create starttime and endtime and snap them down to the 5 minute boundary.
4. Create an earlier time by subtracting one 5-minute time period from starttime and a later time by adding one 5-minute period to endtime
5. Create one big field that you will split into four "events" for graphing - earlier and later with 0s, and starttime and endtime with 1s.
6. Split out the field with makemv, and explode into different events using mvexpand
7. re-create the _time, split-by field and value of 0 or 1 for each "exploded" event
8. Graph them, making sure to select "connect points" as the option for null values!

search:

index=_internal sourcetype=splunkd listening queues *blocked
| transaction host startswith="stopping" endswith="started"
| where duration>=300
| eval starttime=_time 
| bucket starttime span=5m
| eval earlier=starttime-300 
| eval endtime=_time+duration 
| bucket endtime span=5m
| eval later=endtime+300
| eval values=earlier + "," + host + ",0|" + starttime  +  ","  + host  + ",1|" + endtime + "," + host + ",1|" + later + "," + host + ",0"
| table values
| makemv delim="|" values
| mvexpand values
| table values
| rex field=values "(?<_time>[^,]+),(?<host>[^,]+),(?<value>[01])"
| table _time host value
| timechart span=5m max(value) by host

Click image for full size view: