Splunk Search
Highlighted

How to search and alert on unbalanced load across hosts for a target sourcetype?

Explorer

Hello

I am trying to set up a Splunk search which will alert on unbalanced load across hosts for a target sourcetype.

Scenario:
sourcetype="A" has 4 hosts h1, h2, h3 and h4

Trigger alert for unbalanced load whenever load on 1 host is n% less or more than any other host

sourcetype="A" index="*" | timechart span=8h count by host

h1: 100 hits
h2: 95 hits
h3. 91 hits
h4. 85 hits

Alert: h4 is unbalanced by +/- n%

**I have been searching for something similar in this community, haven't found anything yet, any help will be appreciated
Thanks

0 Karma
Highlighted

Re: How to search and alert on unbalanced load across hosts for a target sourcetype?

Contributor

Try this

| appendpipe [streamstats range(count) as difference max(count) as max | eval perc=difference/max*100] | search perc>=1

0 Karma
Highlighted

Re: How to search and alert on unbalanced load across hosts for a target sourcetype?

Contributor

I'll admit this only compares to the max

0 Karma
Highlighted

Re: How to search and alert on unbalanced load across hosts for a target sourcetype?

Esteemed Legend

Like this:

sourcetype="A" index="*" | timechart span=8h count by host
| eventstats min(count) AS least max(count) AS greatest
| eval minPctDiff = count - least * 200 / (count + least)
| eval maxPctDiff = greatest - count * 200 / (count + greatest)
| eval threshold=10 | rename Comment AS "<- put your threshold here!"
| where minPctDiff>threshold OR maxPctDiff>threshold
0 Karma
Highlighted

Re: How to search and alert on unbalanced load across hosts for a target sourcetype?

SplunkTrust
SplunkTrust

This will be little in-efficient but should do the trick. This is assuming your %change threshold is 10%)

sourcetype="A" index="*" | bucket span=8h _time | stats count by _time host | eventstats values(count) as allhosts by _time | mvexpand allhosts | where allhosts>1.1*count OR allhosts<1.1*count
0 Karma
Highlighted

Re: How to search and alert on unbalanced load across hosts for a target sourcetype?

Explorer

Thank you guys for directing me on this search
I gathered ideas from above two answers and put together a query

It calculates target load per host by (total load/host count) then calculates %off (per host) from target load and compares against set threshold which is set to 90 for this search

sourcetype="A" index="B" | stats count by host sourcetype | eventstats avg(count) as avg | eval target=(count/avg)*100 | where target < 90

now i am receiving alerts on un-balanced load for target sourcetype

0 Karma