topic Re: How to edit my search to alert once per result for multiple hosts? in Splunk Search

How to edit my search to alert once per result for multiple hosts?

CaptainHook — Fri, 20 Jan 2017 13:23:10 GMT

We are using Splunk 6.4.2 and I have alerting setup on a specific search as follows:

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN

We would like to be alerted for each event that comes up on the individual host; however, when the alerting happens, it creates multiple events under one alert. Unfortunately, due to the request of only wanting to alert on the specific 10 hosts out of 25 hosts, I have to include the host names in the search. I do have alert mode set to: "Once per Result". Is there something that can be changed so Splunk alerts if any of these hosts events show? As stated, we would like it to be one alert for each event.

Should I change the search or do I need to set up alerting individually for each one? Any suggestions would be greatly appreciated.

Thank you.

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Fri, 20 Jan 2017 15:39:41 GMT

When you edit the alert from Alerts dashboard, on first page, the alert type will be 'scheduled'. Click on next to go to 'Enable action' screen, scroll down to 'Action options' section. Right now it should be 'Once' for 'When triggered execute actions'. Change it to 'For each result'. Please note that if there are more than 1 rows for a host in your resultset, you'll get that many alert. To be sure, I would run some aggregation on your alert search.

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Fri, 20 Jan 2017 15:51:34 GMT

Thank you for your quick response and I do already have it set to trigger "for each result". I am trying to determine if having all the hosts in my search is creating contention in the trigger, as it returns multiple events for different hosts as one event trigger.

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Fri, 20 Jan 2017 16:05:10 GMT

Assuming the search is the exact alert search, try something like this. basically combining the search to merge all events related to a host as one row, so you get one alert per host.

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | transaction host

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | stats values(_raw) as _raw by host

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Fri, 20 Jan 2017 16:10:19 GMT

Thank you, I will try adding that to my search now and will update the post accordingly. I appreciate your expertise and assistance.

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 16:52:21 GMT

Unfortunately, neither of these have worked. Any other suggestions or even different ways to monitor the events individually?

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 17:01:20 GMT

Try this as well

index = wineventlogs 
 sourcetype = wineventlog_sec 
 host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | dedup host

Also, when you say it didn't work, what was happening?

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 18:18:18 GMT

I will try that now.

The alerting works, but it will group all the hosts into one alert. So, the file (pdf) that gets emailed to our support teams will contain multiple events, instead of one pdf (alert) for each host.

The goal is to have one host alert and then a separate ticket is automatically generated to the support teams. Currently, it is opening one ticket for multiple hosts.

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 18:58:15 GMT

Is the alert properly configured to send email per event? When you edit the alert from Alerts dashboard, in the 2nd page, does it say "per event" under Action options section?

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 19:05:25 GMT

It is, yes. It worked prior when I did not include hosts in the search. However, the customer would like to only report on a number of hosts and not all.

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 19:11:27 GMT

So before you added host name in the search, you're getting 25 emails, one for each host? I'm assuming that's what you want now, 10 emails, one for each host?

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 19:17:00 GMT

That is correct. Now, we only want to alert on 10 of the 25 hosts. One email for each host as it occurs. In practice, we want to be alerted whenever someone logs into the host via a specific method.

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 19:23:34 GMT

If the alert was sending one email per host earlier, it should be doing the same, after explicitly selecting fewer hosts. Has anything else changed in the alert? Is the query that you posted the full query? If not, then could you post full alert-search what was before and what you changed into?

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Tue, 29 Sep 2020 12:34:42 GMT

Sorry, let me clarify:
The search originally only looked at index and sourcetype; there was no hosts listed in the search. So anytime we saw events under that sourcetype it would alert.

Since then, the customer decided that they want to be able to search on all the hosts still, but only alert on higher value machines.

So, the original (working) search was:
index = abc_wineventlogs
sourcetype = abc_wineventlog_sec

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 19:39:41 GMT

I'm stumped. The changes that you made should not be causing the "per event" alerting at all. Is this a real-time alert?

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 20:20:31 GMT

Me too 🙂 This is a scheduled search (cron 5 minutes)

Re: How to edit my search to alert once per result for multiple hosts?

somesoni2 — Mon, 23 Jan 2017 20:25:29 GMT

Could you try one thing? Create a new alert search altogether with this new search (with host) and see if that works. Disable the current one though to avoid duplicate alerts.

Re: How to edit my search to alert once per result for multiple hosts?

CaptainHook — Mon, 23 Jan 2017 20:32:17 GMT

I will give that a shot once I hear back from the customer on testing the |dedup host. Updates to follow. Thank you for all your time and suggestions.

Re: How to edit my search to alert once per result for multiple hosts?

sloshburch — Wed, 25 Jan 2017 13:23:04 GMT

This sounds like a classic use of the Alert mode features. Perhaps you have it set up correctly but didn't put the Host in the throttling fields? You'll want to put the unique item in that field so it will alert once per result but then ignore (throttle) if it sees the same one again within the time frame you listed.

Also, what is the full search string? For sanity, we should make sure your search string isn't undermining your alerting.

Re: How to edit my search to alert once per result for multiple hosts?

sloshburch — Wed, 25 Jan 2017 13:31:10 GMT

Crappers. I just noticed this details after I posted my answer.