All Apps and Add-ons

Count overlaping intervals

icegras
Explorer

I have a server with 16 threads executing stuff, and want to detect thread starvation (all threads busy, stopping all other activity). I have logs that get written at the end of every execution, with the execution time. With that I can build a table with threadNumber, start time, end time, and duration.

Using the Timeline visualization plugin, I can chart the activity of all threads as 16 lines that show many horizontal bars representing an execution, and see when that starvation happened (16 simultaneous horizontal bars stacking up), provided I make the time-frame small enough or else I'd get drowned in data. It works, but I can have systems with 32 or more threads, so it gets unreadable quickly.

alt text

I'd like to find a way to get the count of overlaping intervals and maybe chart it, so you'd see the number of busy threads slowly climbing up to 16 and staying there stuck for a few minutes until one of the 16 finally completes and starts picking up the backlog.

Any ideas welcome!

0 Karma
1 Solution

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

View solution in original post

sundareshr
Legend

Have you looked at the concurrency command? Concurrency measures the number of events which have spans that overlap with the start of each event.

https://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Concurrency

icegras
Explorer

Thanks for that command, did not know about it, even after some searching. I get approximate results (sometimes 18 busy threads out of 16. because of bucket size), but it gives you an idea that something is wrong when you are around the 16 mark.
Here is something that worked for me:

search ExecutionTime>2| eval _time = _time - round(ExecutionTime) | concurrency duration=ExecutionTime | timechart max(concurrency) span=1m

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...