topic Re: Create alert when volume of logs increases significantly from a particular host in Alerting

Create alert when volume of logs increases significantly from a particular host

mbond_illumina — Wed, 26 Jun 2013 14:36:35 GMT

I have a problem with a server that keeps violating the splunk indexing volume for the day as the volume of it's logs increases hugely. I'd like to set an alert so that when a theshold is reached it sends out an alert so I can investigate.

Re: Create alert when volume of logs increases significantly from a particular host

sdaniels — Wed, 26 Jun 2013 15:34:24 GMT

You can use a search such as this to look at volume for a particular host and then create an alert off of the search based on a threshold. This should look at volume of data in MB from "yourhostname" over the last 2 hours for example.

index=_internal source=*metrics.log series="<yourhostname>" earliest=-2h | eval MB=kb/1024 | chart sum(MB) by series | sort -sum(MB)

If you want something a bit more complex you could look at using a standard deviation calculation similar to this:

http://splunk-base.splunk.com/answers/47920/comparing-standard-deviations

Re: Create alert when volume of logs increases significantly from a particular host

mbond_illumina — Wed, 26 Jun 2013 16:06:25 GMT

Thanks for you reply. I was really looking at counting the number of logs as opposed to the amount in MB. Is it possible to do that? I did try the example you gave but it didn't give me a value for the MB for some reason.

Re: Create alert when volume of logs increases significantly from a particular host

sdaniels — Wed, 26 Jun 2013 18:24:47 GMT

You could just do a search on the number of events (not logs) but essentially you have individual events coming in from those logs so it's similar. Here is an example search that gives me the count of events on one host in the last 2 hours.

sourcetype="access_combined" host="apache-1.splunk.com" earliest=-2h| stats count

Re: Create alert when volume of logs increases significantly from a particular host

jtrucks — Wed, 26 Jun 2013 19:18:09 GMT

You could use two approaches, volume of data and number of log entries.

Volume of data the empirical way:

earliest=-0d@d host=www* | eval b=len(_raw) | eval MB=b/1024/1024 | stats sum(MB)

Then alert based on a specific threshold. This may be fast enough to alert on it if you run this every few minutes to give it time to count, especially when it gets later in the day.

Number of log entries:

earliest=-0d@d host=www* | stats count

Then alert on a specific threshold.

If you want to alert based on rate of change, you get to use the above searches with a timechart instead of stats and do some type of comparison. This gets into funky nested searches and comparisons, possibly even summary indexes to compare previous data to current data etc. That may be overkill for what you want to do, however.

Re: Create alert when volume of logs increases significantly from a particular host

kristian_kolb — Wed, 26 Jun 2013 19:22:32 GMT

or if you have a laaarge amount of data and don't want to search it unnecessarily, you could probably combine inputlookup/outputlookup with a metadata search in between to append to the lookup...

Or you can install the Splunk Depolyment Monitor app.

Re: Create alert when volume of logs increases significantly from a particular host

jtrucks — Wed, 26 Jun 2013 19:23:16 GMT

Another thing to consider is throttling that machine's logging through syslog or the indexer or a forwarder in the middle.

Re: Create alert when volume of logs increases significantly from a particular host

kristian_kolb — Wed, 26 Jun 2013 19:58:23 GMT

well. throttling the forwarder will reduce the load on the indexer and conserve license space, but it will fill up queues on the forwarder... not really any better. potential loss of events.

Re: Create alert when volume of logs increases significantly from a particular host

mbond_illumina — Thu, 27 Jun 2013 11:21:59 GMT

Thanks to you both for your answers. That works great. However, when I create my alert based on the search, what do I choose for the 'Trigger If' setting? I presume I need to enter a custom condition. Do I enter 'MB > 100?'? If I chose 'No of Results' then wouldn't that just be the number of matched events rather than the Sum(MB)?

Re: Create alert when volume of logs increases significantly from a particular host

jtrucks — Thu, 27 Jun 2013 16:10:49 GMT

Do saved search like this for better output:

host=www* | eval b=len(_raw) | eval subMB=b/1024/1024 | stats sum(subMB) as MB

This way MB is the sum total, not the individual event data. Then set Time Range to:

Start time: -0d@d Finish time: now

Then Alert Condition custom:

where MB > 100

I had it send results via email and inline and the result is a table that just has:

   MB
187.343

Set the frequency to anything you want, like every 5 minutes or whatever works for you. If it takes more than 60 seconds to complete the search near the end of the day, don't do it every minute.

Re: Create alert when volume of logs increases significantly from a particular host

jtrucks — Thu, 27 Jun 2013 16:14:05 GMT

Also, please mark Answered if this works for you 🙂

Re: Create alert when volume of logs increases significantly from a particular host

jtrucks — Thu, 11 Jul 2013 15:44:01 GMT

The second option in my answer earlier shows how to count events, as well.