Splunk Search
Highlighted

Plot up or down state over time?

Motivator

I have a dataset I just created using transaction that shows when a particular service is down by pulling in the "service down" message and the "service up" message.

How do I plot when the service is up vs down on a graph over time?

Similar to the question here.

Highlighted

Re: Plot up or down state over time?

Motivator

It just so happens the important service is my splunk indexing, and I'm tracking when the queues become so full the network ports are shut off.

I found a way to do this, but it requires a set time range, because you have to hack _time a few times in the search. The idea is to turn the single "on" events into four events - the beginning and end times of the "on" status - and the "off" status just before and after the "on". Then use "connect points" to create the shape!

The idea is this:

  1. Create the transactions, or pull out some sort of events that you can create a starttime and endtime from.
  2. In this case, I used periods when the indexer was unable to receive data because its network port was closed due to over 5 minutes of blocked queues. Transaction gives me _time and _time + duration to use.
  3. Create starttime and endtime and snap them down to the 5 minute boundary.
  4. Create an earlier time by subtracting one 5-minute time period from starttime and a later time by adding one 5-minute period to endtime
  5. Create one big field that you will split into four "events" for graphing - earlier and later with 0s, and starttime and endtime with 1s.
  6. Split out the field with makemv, and explode into different events using mvexpand
  7. re-create the _time, split-by field and value of 0 or 1 for each "exploded" event
  8. Graph them, making sure to select "connect points" as the option for null values!

search:

index=_internal sourcetype=splunkd listening queues *blocked
| transaction host startswith="stopping" endswith="started"
| where duration>=300
| eval starttime=_time 
| bucket starttime span=5m
| eval earlier=starttime-300 
| eval endtime=_time+duration 
| bucket endtime span=5m
| eval later=endtime+300
| eval values=earlier + "," + host + ",0|" + starttime  +  ","  + host  + ",1|" + endtime + "," + host + ",1|" + later + "," + host + ",0"
| table values
| makemv delim="|" values
| mvexpand values
| table values
| rex field=values "(?<_time>[^,]+),(?<host>[^,]+),(?<value>[01])"
| table _time host value
| timechart span=5m max(value) by host

Click image for full size view:

Up Down Status Graphs