Splunk Search

How to search and alert on unbalanced load across hosts for a target sourcetype?

snabi
Explorer

Hello

I am trying to set up a Splunk search which will alert on unbalanced load across hosts for a target sourcetype.

Scenario:
sourcetype="A" has 4 hosts h1, h2, h3 and h4

Trigger alert for unbalanced load whenever load on 1 host is n% less or more than any other host

sourcetype="A" index="*" | timechart span=8h count by host

h1: 100 hits
h2: 95 hits
h3. 91 hits
h4. 85 hits

Alert: h4 is unbalanced by +/- n%

**I have been searching for something similar in this community, haven't found anything yet, any help will be appreciated
Thanks

0 Karma

snabi
Explorer

Thank you guys for directing me on this search
I gathered ideas from above two answers and put together a query

It calculates target load per host by (total load/host count) then calculates %off (per host) from target load and compares against set threshold which is set to 90 for this search

sourcetype="A" index="B" | stats count by host sourcetype | eventstats avg(count) as avg | eval target=(count/avg)*100 | where target < 90

now i am receiving alerts on un-balanced load for target sourcetype

0 Karma

somesoni2
Revered Legend

This will be little in-efficient but should do the trick. This is assuming your %change threshold is 10%)

sourcetype="A" index="*" | bucket span=8h _time | stats count by _time host | eventstats values(count) as allhosts by _time | mvexpand allhosts | where allhosts>1.1*count OR allhosts<1.1*count
0 Karma

woodcock
Esteemed Legend

Like this:

sourcetype="A" index="*" | timechart span=8h count by host
| eventstats min(count) AS least max(count) AS greatest
| eval minPctDiff = count - least * 200 / (count + least)
| eval maxPctDiff = greatest - count * 200 / (count + greatest)
| eval threshold=10 | rename Comment AS "<- put your threshold here!"
| where minPctDiff>threshold OR maxPctDiff>threshold
0 Karma

dolivasoh
Contributor

Try this

| appendpipe [streamstats range(count) as difference max(count) as max | eval perc=difference/max*100] | search perc>=1

0 Karma

dolivasoh
Contributor

I'll admit this only compares to the max

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...