topic Re: Creating alarms based on differences in stats output in Splunk Search

Creating alarms based on differences in stats output

sknot1454 — Thu, 21 Aug 2014 16:17:41 GMT

Greetings!

Right now we're monitoring connections between internal IPs and external IPs using our proxy log input.

We monitor total bytes sent, average bytes sent, mode bytes sent, STDVE bytes sent, var bytes sent, and range bytes sent using the Eventstats search.

This report is ran every hour.

What I would like to do is set up an alarm if there's some kind of statistical anomaly regarding the data being sent. For example, Host 1 averages 10mb of traffic every hour over HTTP. Host 1 becomes compromised and there's a massive data exfil that sends the average per hour to 1GB.

How can we setup some kind threshold to alert us of a massive deviation from the normal range?

Thanks!

Re: Creating alarms based on differences in stats output

starcher — Thu, 21 Aug 2014 17:09:31 GMT

I would recommend watching Jesse Trucks on trending and stddev etc.
http://vimeo.com/66779015

Re: Creating alarms based on differences in stats output

sknot1454 — Thu, 21 Aug 2014 18:15:17 GMT

Informative video, thanks.

Unfortunately, he's looking at it from a much higher view than I am. I don't care about total events generated by a sourcetype. I care more about trending a single field value e.g "bytes sent" and the deviation from that.

Re: Creating alarms based on differences in stats output

somesoni2 — Thu, 21 Aug 2014 18:53:22 GMT

My be something like this will work.

search/stats to get "byte sent" for current hour per host | table host,byteSentCurrent ##| join host [##search/stats to## get avg "byte sent" for past 1 day or any other period per host | table host, avgByteSent##] | ##compare percent difference between byteSentCurrent and avgByteSent and alert based on that

Re: Creating alarms based on differences in stats output

sknot1454 — Fri, 22 Aug 2014 13:19:05 GMT

I think I have a decent solution now.

I'm just going to have to create a query that looks at avg(byes_sent) per minute for a particular host type(web server,DC,IIS, etc). Monitor that query every day for like a week and drill down a predictable average and calculate the STDEV from that.

Based off of that STDEV, I can create a search query with a WHERE statement that says "return results where STDEV > baseline". If the query brings back any results, fire off an alarm.

Thanks again!

Re: Creating alarms based on differences in stats output

prelert — Mon, 28 Sep 2020 17:28:31 GMT

A straightforward solution would be to run:

... | prelertautodetect sum(bytes_sent) by host_type

This would baseline the total bytes sent from each host_type (accounting for periodicity and behaviour not well described by STDEV and MEAN etc.) and create an anomaly where a specific host_type sends unusual volumes of data.

An issue with this analysis is that if there are a large number of hosts of a particular host_type, then a deviation of one host may be lost in the aggregation.

Therefore, an extension to this could be to partition the hosts by type and then analyse each host in that partition. For example,

... | prelertautodetect partitionfield=host_type sum(bytes_sent) over host

In this analysis, a statistical profile is created for each host_type and each host is compared to this profile.

All these searches can be run continuously in real-time.