Splunk Search

Real-Time Search and Alerting

mknowles
Engager

Hello,

I've figured out how to start a real-time search job. I'm wondering if there's any way to trigger a shell command or generate an email/alert every time a new event appears in the real-time search output?

For example, how would I go about getting an email everytime somebody logs on to a server as Administrator (in real-time)?

Thanks, Mark

Tags (1)

rashidmirza
New Member

is there any documentation on how splunk v4.2 supports real time alerting? like a step by step procedure on how splunk v4.2 can be configured for a real time task?

0 Karma

Jason
Motivator

Splunk v4.2 now supports real-time alerting.

Lowell
Super Champion

To add what what gkanapathy said, you may be able to use a tool such as the Simple Event Coorelator to handle something like this. SEC can read from just about any file or pipe and can be setup to trigger on a simple or complex of events that you want; so you could easily pipe the output from a splunk search into SEC.

I have to admit that for me, this does feel like a step backwards. We've used SEC to monitor log files and trigger events before I had even heard of splunk, and now I've removed most of the processing rules we made for SEC and migrated most of that pattern matching logic into Splunk. Generally speaking, Splunk it's much easier to manage, easier to navigate, and provides massive visibility and flexibility improvements over what we has setup with SEC.

However, with that said, we still do use SEC for some things that Splunk can't do yet. For example, trigger a firewall blacklisting script after so many consecutive failed FTP logins. This could somewhat be accomplished with splunk, but we would be looking at a 1-2 minute gap between attack and blacklist. (We'd also have to setup a call back feature between our central splunk indexer/search head and the forwarder machine.) Whereas with SEC everything is local, and the attack gets shutdown in a few seconds.

I'm really hoping that as splunk progresses in the real-time search features, this kind of functionality will start to become possible, and even ideally, handled from within splunk.

But in the meantime, such a tool might be helpful for you.

gkanapathy
Splunk Employee
Splunk Employee

The simple answer is that there really isn't a way to do real-time alerting in 4.1.x, and won't be until a later release. The more complicated answer is that if you are motivated enough, you can put something together using real-time search at the command line that pipes to another simple script that sends an alert every time the real-time search outputs a line. I admit that I find it a bit hacky, but that's the best I can think of right now.

netwrkr
Communicator

netwrkr
Communicator

Well, you can schedule a search to run every minute.

0 Karma

mknowles
Engager

Hi netwrkr, thanks for the response. That page only seems to apply to scheduled searches, not real time searches. Is the only way to do alerting with scheduled searches? Ie schedule it every minute or something?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...