Processing events with an external script

[wibay]

I'd like to take various actions against real-time events from Splunk. What's considering the best practice for this? I'm not sure if it would be preferred to run a real-time search in Splunk and kick off a scripted alert for those events that I want to process further (running zcat on the $8 argument to the script to get at the events), or if it's better to run a "splunk rtsearch" from the CLI. Any pointers on how to really iterate through the rtsearch results, similar to something like this:

splunk rtsearch "link down | table _time, host" -preview false -header false | while read time host; do
  something
done

[woodcock]

I would advise against actual real-time searches and use near-real-time scheduled searches instead. Unless you have designed/scaled your infrastructure explicitly for rtsearch, you will become a very disliked person because each rtsearch permanently consumes 1 core on each Splunk server. Using rtsearch also has the problem that the smaller window you use, the more events you will never see due to various pipeline latencies (e.g. if the event does not arrive in Splunk until 2 minutes after it happened and you have a 1-minute window, you will never see the event). What you really need to do is schedule a search on a duty cycle that is longer than your biggest average delivery latency. So something like every 5 minutes for the last 5 minutes. In the schedule options, you can set a script to call when the results set has more than 1 event. Now all you need to do is write the script and put it on your search head.