Real-Time Search and Alerting

[mknowles]

Hello,

I've figured out how to start a real-time search job. I'm wondering if there's any way to trigger a shell command or generate an email/alert every time a new event appears in the real-time search output?

For example, how would I go about getting an email everytime somebody logs on to a server as Administrator (in real-time)?

Thanks, Mark

[rashidmirza]

is there any documentation on how splunk v4.2 supports real time alerting? like a step by step procedure on how splunk v4.2 can be configured for a real time task?

[Jason]

Splunk v4.2 now supports real-time alerting.

[Lowell]

To add what what gkanapathy said, you may be able to use a tool such as the Simple Event Coorelator to handle something like this. SEC can read from just about any file or pipe and can be setup to trigger on a simple or complex of events that you want; so you could easily pipe the output from a splunk search into SEC.

I have to admit that for me, this does feel like a step backwards. We've used SEC to monitor log files and trigger events before I had even heard of splunk, and now I've removed most of the processing rules we made for SEC and migrated most of that pattern matching logic into Splunk. Generally speaking, Splunk it's much easier to manage, easier to navigate, and provides massive visibility and flexibility improvements over what we has setup with SEC.

However, with that said, we still do use SEC for some things that Splunk can't do yet. For example, trigger a firewall blacklisting script after so many consecutive failed FTP logins. This could somewhat be accomplished with splunk, but we would be looking at a 1-2 minute gap between attack and blacklist. (We'd also have to setup a call back feature between our central splunk indexer/search head and the forwarder machine.) Whereas with SEC everything is local, and the attack gets shutdown in a few seconds.

I'm really hoping that as splunk progresses in the real-time search features, this kind of functionality will start to become possible, and even ideally, handled from within splunk.

But in the meantime, such a tool might be helpful for you.

[gkanapathy]

The simple answer is that there really isn't a way to do real-time alerting in 4.1.x, and won't be until a later release. The more complicated answer is that if you are motivated enough, you can put something together using real-time search at the command line that pipes to another simple script that sends an alert every time the real-time search outputs a line. I admit that I find it a bit hacky, but that's the best I can think of right now.

[netwrkr]

Check this out -

http://www.splunk.com/base/Documentation/latest/User/SetAlertConditionsFromScheduledSearches

[netwrkr]

Well, you can schedule a search to run every minute.

[mknowles]

Hi netwrkr, thanks for the response. That page only seems to apply to scheduled searches, not real time searches. Is the only way to do alerting with scheduled searches? Ie schedule it every minute or something?