topic Re: Real-Time Search and Alerting in Splunk Search

Real-Time Search and Alerting

mknowles — Tue, 18 May 2010 15:33:49 GMT

Hello,

I've figured out how to start a real-time search job. I'm wondering if there's any way to trigger a shell command or generate an email/alert every time a new event appears in the real-time search output?

For example, how would I go about getting an email everytime somebody logs on to a server as Administrator (in real-time)?

Thanks, Mark

Re: Real-Time Search and Alerting

netwrkr — Wed, 19 May 2010 02:48:00 GMT

Check this out -

http://www.splunk.com/base/Documentation/latest/User/SetAlertConditionsFromScheduledSearches

Re: Real-Time Search and Alerting

mknowles — Wed, 19 May 2010 14:12:45 GMT

Hi netwrkr, thanks for the response. That page only seems to apply to scheduled searches, not real time searches. Is the only way to do alerting with scheduled searches? Ie schedule it every minute or something?

Re: Real-Time Search and Alerting

gkanapathy — Wed, 19 May 2010 16:30:05 GMT

The simple answer is that there really isn't a way to do real-time alerting in 4.1.x, and won't be until a later release. The more complicated answer is that if you are motivated enough, you can put something together using real-time search at the command line that pipes to another simple script that sends an alert every time the real-time search outputs a line. I admit that I find it a bit hacky, but that's the best I can think of right now.

Re: Real-Time Search and Alerting

netwrkr — Wed, 19 May 2010 19:40:33 GMT

Well, you can schedule a search to run every minute.

Re: Real-Time Search and Alerting

Lowell — Fri, 21 May 2010 06:18:07 GMT

To add what what gkanapathy said, you may be able to use a tool such as the Simple Event Coorelator to handle something like this. SEC can read from just about any file or pipe and can be setup to trigger on a simple or complex of events that you want; so you could easily pipe the output from a splunk search into SEC.

I have to admit that for me, this does feel like a step backwards. We've used SEC to monitor log files and trigger events before I had even heard of splunk, and now I've removed most of the processing rules we made for SEC and migrated most of that pattern matching logic into Splunk. Generally speaking, Splunk it's much easier to manage, easier to navigate, and provides massive visibility and flexibility improvements over what we has setup with SEC.

However, with that said, we still do use SEC for some things that Splunk can't do yet. For example, trigger a firewall blacklisting script after so many consecutive failed FTP logins. This could somewhat be accomplished with splunk, but we would be looking at a 1-2 minute gap between attack and blacklist. (We'd also have to setup a call back feature between our central splunk indexer/search head and the forwarder machine.) Whereas with SEC everything is local, and the attack gets shutdown in a few seconds.

I'm really hoping that as splunk progresses in the real-time search features, this kind of functionality will start to become possible, and even ideally, handled from within splunk.

But in the meantime, such a tool might be helpful for you.

Re: Real-Time Search and Alerting

Jason — Sat, 02 Apr 2011 01:14:11 GMT

Splunk v4.2 now supports real-time alerting.

Re: Real-Time Search and Alerting

rashidmirza — Thu, 20 Oct 2011 09:41:34 GMT

is there any documentation on how splunk v4.2 supports real time alerting? like a step by step procedure on how splunk v4.2 can be configured for a real time task?