Splunk Search

How to search and alert on unbalanced load across hosts for a target sourcetype?

snabi
Explorer

Hello

I am trying to set up a Splunk search which will alert on unbalanced load across hosts for a target sourcetype.

Scenario:
sourcetype="A" has 4 hosts h1, h2, h3 and h4

Trigger alert for unbalanced load whenever load on 1 host is n% less or more than any other host

sourcetype="A" index="*" | timechart span=8h count by host

h1: 100 hits
h2: 95 hits
h3. 91 hits
h4. 85 hits

Alert: h4 is unbalanced by +/- n%

**I have been searching for something similar in this community, haven't found anything yet, any help will be appreciated
Thanks

0 Karma

snabi
Explorer

Thank you guys for directing me on this search
I gathered ideas from above two answers and put together a query

It calculates target load per host by (total load/host count) then calculates %off (per host) from target load and compares against set threshold which is set to 90 for this search

sourcetype="A" index="B" | stats count by host sourcetype | eventstats avg(count) as avg | eval target=(count/avg)*100 | where target < 90

now i am receiving alerts on un-balanced load for target sourcetype

0 Karma

somesoni2
Revered Legend

This will be little in-efficient but should do the trick. This is assuming your %change threshold is 10%)

sourcetype="A" index="*" | bucket span=8h _time | stats count by _time host | eventstats values(count) as allhosts by _time | mvexpand allhosts | where allhosts>1.1*count OR allhosts<1.1*count
0 Karma

woodcock
Esteemed Legend

Like this:

sourcetype="A" index="*" | timechart span=8h count by host
| eventstats min(count) AS least max(count) AS greatest
| eval minPctDiff = count - least * 200 / (count + least)
| eval maxPctDiff = greatest - count * 200 / (count + greatest)
| eval threshold=10 | rename Comment AS "<- put your threshold here!"
| where minPctDiff>threshold OR maxPctDiff>threshold
0 Karma

dolivasoh
Contributor

Try this

| appendpipe [streamstats range(count) as difference max(count) as max | eval perc=difference/max*100] | search perc>=1

0 Karma

dolivasoh
Contributor

I'll admit this only compares to the max

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...