All Apps and Add-ons

Count overlaping intervals

icegras
Explorer

I have a server with 16 threads executing stuff, and want to detect thread starvation (all threads busy, stopping all other activity). I have logs that get written at the end of every execution, with the execution time. With that I can build a table with threadNumber, start time, end time, and duration.

Using the Timeline visualization plugin, I can chart the activity of all threads as 16 lines that show many horizontal bars representing an execution, and see when that starvation happened (16 simultaneous horizontal bars stacking up), provided I make the time-frame small enough or else I'd get drowned in data. It works, but I can have systems with 32 or more threads, so it gets unreadable quickly.

alt text

I'd like to find a way to get the count of overlaping intervals and maybe chart it, so you'd see the number of busy threads slowly climbing up to 16 and staying there stuck for a few minutes until one of the 16 finally completes and starts picking up the backlog.

Any ideas welcome!

0 Karma
1 Solution

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

View solution in original post

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

View solution in original post

icegras
Explorer

Thanks for that command, did not know about it, even after some searching. I get approximate results (sometimes 18 busy threads out of 16. because of bucket size), but it gives you an idea that something is wrong when you are around the 16 mark.
Here is something that worked for me:

search ExecutionTime>2| eval _time = _time - round(ExecutionTime) | concurrency duration=ExecutionTime | timechart max(concurrency) span=1m

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!