Getting Data In

monitor that proper amount of events are coming in per host

jfraiberg
Communicator

I am trying to build an app that will set a baseline per host of event count that will alert me when a hosts event count has dropped by a certain % per hour.

Has anyone tried something like this? I find that monitoring the agent alone is not enough and would like to know if a host is sending less events than normal.

Any ideas or advice would be welcome 🙂

Tags (1)
0 Karma
1 Solution

ziegfried
Influencer

This search would compare the count by host of the previous hour with the current hour and filter those where the count dropped by more than 10%:

earliest=-2h@h latest=@h | stats count by date_hour,host | stats first(count) as prev last(count) as cur by host | eval cur_pct=(cur/prev)*100 | where cur_pct<90

View solution in original post

ziegfried
Influencer

This search would compare the count by host of the previous hour with the current hour and filter those where the count dropped by more than 10%:

earliest=-2h@h latest=@h | stats count by date_hour,host | stats first(count) as prev last(count) as cur by host | eval cur_pct=(cur/prev)*100 | where cur_pct<90

jfraiberg
Communicator

This totally worked, thanks! The search itself took forever though and took alot of resources to run. I will play with it some more. thanks!!!

0 Karma

jameswatts
Explorer

You can speed the search up significantly with tstats:
|tstats count by host, _time span=1h| stats first(count) as prev last(count) as cur by host | eval cur_pct=(cur/prev)*100 | where cur_pct<90

0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...