Can alert condition be set in command line search

[alextsui]

Hi. How would I run a search command in command line. The problem is that I would also like to set an alert condition like I could with a savedsearch. For example, if the returned search results contains more than 3 events, an alert is triggered to send out an email (or execute a script if possible).

Thanks.

[Lowell]

The alerting condition functionality is based on the scheduler so you can't get the exact same functionality with a normal interactive search that you can with the scheduler-- it doesn't matter if your interactive search is via the web interface, command line search, or via the "REST" (programmatic) interface.

That said, behind the scenes, all of the alert actions are really just search commands anyways, so you can get nearly the same functionality by calling search command directly. (You can see the alert condition mapping by poking around in $SPLUNK_HOME/etc/system/default/alert_actions.conf, look at the "command" setting.) One limitation is that you will not be able to use multiple "actions" with an interactive search like you could with a saved search.)

Going along with your example situation... You could send an email if you have more than 3 matching events by doing something like this:

error OR fail* | stats count as events by host, source | eventstats count as event_count | where event_count>3 | sendemail to=joe@example.com from=splunk-alert@example.com subject="A sample alert" sendresults=true format=html inline=true

Update: As pointed out in the comments, this approach doesn't actually work. In the sense that if there are less than 3 events an email is still sent, but it contains no results. (The email body literally says "Saved search results." (which isn't true), followed by the phrase "No results.") So this doesn't actually work.

I'm curious if there is a better answer to this, so I've posted my own question as well:

Can a search be terminated prematurely based on a condition established within that search?

---

Here are some other suggestions:

1. Use a savedsearch, setup an alert condition and schedule it to run. This is easy and well understood.
2. Create a savedsearch and setup a alert condition but do not schedule it. You can make an REST API call that will run a savedsearch immediately and you can pass in a parameter to force the alert actions to be evaluated and the actions to be executed by setting triggerActions=1.
3. If your search has to be dynamic, then use the REST API to create a temporary (one-time use) savedsearch, and then run it immediately (same as in #2), then remove the temporary savedsearch via the API. That's a pain, but it should work.

[Lowell]

Whoops. You are correct. I think you would need some sort of search command that would evaluate a condition, and based on that condition either allow or prevent subsequent search commands from running, but the more I think about it I'm wondering if that's even possible to do with a custom search command. You could probably raise an exception, but that would not be ideal.

[alextsui]

The search query suggested would trigger the email even when the event_count is less than the value specified and contain the message saying no results found. Can the search be modified to trigger the email if, and only if, when the event_count is greater then 3?