Splunk Search

Creating alarms based on differences in stats output

sknot1454
Explorer

Greetings!

Right now we're monitoring connections between internal IPs and external IPs using our proxy log input.

We monitor total bytes sent, average bytes sent, mode bytes sent, STDVE bytes sent, var bytes sent, and range bytes sent using the Eventstats search.

This report is ran every hour.

What I would like to do is set up an alarm if there's some kind of statistical anomaly regarding the data being sent. For example, Host 1 averages 10mb of traffic every hour over HTTP. Host 1 becomes compromised and there's a massive data exfil that sends the average per hour to 1GB.

How can we setup some kind threshold to alert us of a massive deviation from the normal range?

Thanks!

Tags (2)
0 Karma

prelert
Path Finder

A straightforward solution would be to run:

... | prelertautodetect sum(bytes_sent) by host_type

This would baseline the total bytes sent from each host_type (accounting for periodicity and behaviour not well described by STDEV and MEAN etc.) and create an anomaly where a specific host_type sends unusual volumes of data.

An issue with this analysis is that if there are a large number of hosts of a particular host_type, then a deviation of one host may be lost in the aggregation.

Therefore, an extension to this could be to partition the hosts by type and then analyse each host in that partition. For example,

... | prelertautodetect partitionfield=host_type sum(bytes_sent) over host

In this analysis, a statistical profile is created for each host_type and each host is compared to this profile.

All these searches can be run continuously in real-time.

0 Karma

sknot1454
Explorer

I think I have a decent solution now.

I'm just going to have to create a query that looks at avg(byes_sent) per minute for a particular host type(web server,DC,IIS, etc). Monitor that query every day for like a week and drill down a predictable average and calculate the STDEV from that.

Based off of that STDEV, I can create a search query with a WHERE statement that says "return results where STDEV > baseline". If the query brings back any results, fire off an alarm.

Thanks again!

0 Karma

somesoni2
Revered Legend

My be something like this will work.

search/stats to get "byte sent" for current hour per host | table host,byteSentCurrent ##| join host [##search/stats to## get avg "byte sent" for past 1 day or any other period per host | table host, avgByteSent##] | ##compare percent difference between byteSentCurrent and avgByteSent and alert based on that

0 Karma

sknot1454
Explorer

Informative video, thanks.

Unfortunately, he's looking at it from a much higher view than I am. I don't care about total events generated by a sourcetype. I care more about trending a single field value e.g "bytes sent" and the deviation from that.

0 Karma

starcher
SplunkTrust
SplunkTrust

I would recommend watching Jesse Trucks on trending and stddev etc.
http://vimeo.com/66779015

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...