All Apps and Add-ons

Count overlaping intervals

icegras
Explorer

I have a server with 16 threads executing stuff, and want to detect thread starvation (all threads busy, stopping all other activity). I have logs that get written at the end of every execution, with the execution time. With that I can build a table with threadNumber, start time, end time, and duration.

Using the Timeline visualization plugin, I can chart the activity of all threads as 16 lines that show many horizontal bars representing an execution, and see when that starvation happened (16 simultaneous horizontal bars stacking up), provided I make the time-frame small enough or else I'd get drowned in data. It works, but I can have systems with 32 or more threads, so it gets unreadable quickly.

alt text

I'd like to find a way to get the count of overlaping intervals and maybe chart it, so you'd see the number of busy threads slowly climbing up to 16 and staying there stuck for a few minutes until one of the 16 finally completes and starts picking up the backlog.

Any ideas welcome!

0 Karma
1 Solution

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

View solution in original post

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

icegras
Explorer

Thanks for that command, did not know about it, even after some searching. I get approximate results (sometimes 18 busy threads out of 16. because of bucket size), but it gives you an idea that something is wrong when you are around the 16 mark.
Here is something that worked for me:

search ExecutionTime>2| eval _time = _time - round(ExecutionTime) | concurrency duration=ExecutionTime | timechart max(concurrency) span=1m

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...