Splunk Search

Histogram/Chart Question

Explorer

Greetings, I am struggling to create a chart to show when our backups begin and end for each server. The purpose is to show how many over lapping backups are running at a single time so that we can stagger them as not to overload our NAS.

We are indexing messages for backups as such:

When it starts: host=hostname name=backup action=begin
When it ends: host=hostname name=backup action=end

I would like something similar to

Hostname:         Time
Host1:          [======]
Host2:        [=====]
Host3:                     [===]
etc.

That is my vision - a nice simple way to see when backups start/end for each host.

Any help or a point in the right direction would be much appreciated.

Thanks

Explorer

I wanted to do something similar. So very easy to to in HTML - and common. Wonder why this is such a roadblock for Splunk.
Thanks

0 Karma

Splunk Employee
Splunk Employee

You probably want to combine transaction with concurrency

... | transaction host name startswith=("action=begin") endswith=("action=end") | concurrency duration=duration

This will list your backups, each with an additional field concurrency indicating the number of backups running at the start of that backup

Splunk Employee
Splunk Employee

Oh I see, you want a concurrency chart like Gantt-type chart. Unfortunately, Splunk's charting modules don't display these easily with Splunk's data, and I haven't been able to come up with a good way to make it work. I suppose I'd just file an enhancement request. Note that Splunk's own dbinspect command displays a chart like what you want, but it does some ugly hacking to generate data to fit the display capabilities of the Splunk charting modules.

0 Karma

Explorer

I do see the duration. However when I graph the duration, it is graphing it as a value, not time. Is there a way to graph the duration over time? For example, action=begin would be at 1am and action=end would be at 2am. I would like to graph between 1am to 2am for host1, whereas host2 would be from 1:30am-2am.

0 Karma

Splunk Employee
Splunk Employee

the transaction command adds a duration field to each transaction it assembles. Is that not what you need to see? So every transaction group will have _time and duration fields.

0 Karma

Explorer

Excellent! Thank you! I now have the duration of the events. Is there a way to have the duration show at the time that the event started? I am close with:

... | transaction host name startswith=("action=begin") endswith=("action=end") | concurrency duration=duration | timechart span=10m sum(duration) by host

This gets me close but the duration does not match up with the time. Ideas?

Thank you for your time!

0 Karma