We are using Splunk 6.4.2 and I have alerting setup on a specific search as follows:
index = wineventlogs
sourcetype = wineventlog_sec
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN
We would like to be alerted for each event that comes up on the individual host; however, when the alerting happens, it creates multiple events under one alert. Unfortunately, due to the request of only wanting to alert on the specific 10 hosts out of 25 hosts, I have to include the host names in the search. I do have alert mode set to: "Once per Result". Is there something that can be changed so Splunk alerts if any of these hosts events show? As stated, we would like it to be one alert for each event.
Should I change the search or do I need to set up alerting individually for each one? Any suggestions would be greatly appreciated.
Thank you.
I was able to get the customer online to test and none of the changes we made affected the alerting to fire one per event. I went ahead and re-created the alerting with a fresh copy and we are getting better results now. Seems we just had a corrupt alert.
Thank you for all the suggestions and assistance.
I was able to get the customer online to test and none of the changes we made affected the alerting to fire one per event. I went ahead and re-created the alerting with a fresh copy and we are getting better results now. Seems we just had a corrupt alert.
Thank you for all the suggestions and assistance.
I converted this to an answer so you can mark this one as accepted. https://www.youtube.com/watch?v=h-wfO6_eq-A
I appreciate the added self-help link. 🙂
This sounds like a classic use of the Alert mode features. Perhaps you have it set up correctly but didn't put the Host in the throttling fields? You'll want to put the unique item in that field so it will alert once per result but then ignore (throttle) if it sees the same one again within the time frame you listed.
Also, what is the full search string? For sanity, we should make sure your search string isn't undermining your alerting.
Thanks Burch. Yes, we do not want to throttle as they need to see all events (logins) from each host.
It sounds like perhaps the customer is responsible for a subset of the hosts that are contained in the result set. For example, the search may produce alerts for hosts host1, host2, host 3, and host4 but this customer wants one alert when either host1 or host3 trigger.
If that's accurate, then you can create a flag (if you will) for their hosts and then use that flag as the suppression:
eval flag = if( host=="host1" , "annoying_customer", "pleasant_customer" )
Make sense or did I still not understand?
Assuming the search is the exact alert search, try something like this. basically combining the search to merge all events related to a host as one row, so you get one alert per host.
index = wineventlogs
sourcetype = wineventlog_sec
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | transaction host
OR
index = wineventlogs
sourcetype = wineventlog_sec
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | stats values(_raw) as _raw by host
Thank you, I will try adding that to my search now and will update the post accordingly. I appreciate your expertise and assistance.
Unfortunately, neither of these have worked. Any other suggestions or even different ways to monitor the events individually?
Try this as well
index = wineventlogs
sourcetype = wineventlog_sec
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | dedup host
Also, when you say it didn't work, what was happening?
I will try that now.
The alerting works, but it will group all the hosts into one alert. So, the file (pdf) that gets emailed to our support teams will contain multiple events, instead of one pdf (alert) for each host.
The goal is to have one host alert and then a separate ticket is automatically generated to the support teams. Currently, it is opening one ticket for multiple hosts.
Is the alert properly configured to send email per event? When you edit the alert from Alerts dashboard, in the 2nd page, does it say "per event" under Action options section?
It is, yes. It worked prior when I did not include hosts in the search. However, the customer would like to only report on a number of hosts and not all.
So before you added host name in the search, you're getting 25 emails, one for each host? I'm assuming that's what you want now, 10 emails, one for each host?
That is correct. Now, we only want to alert on 10 of the 25 hosts. One email for each host as it occurs. In practice, we want to be alerted whenever someone logs into the host via a specific method.
If the alert was sending one email per host earlier, it should be doing the same, after explicitly selecting fewer hosts. Has anything else changed in the alert? Is the query that you posted the full query? If not, then could you post full alert-search what was before and what you changed into?
Sorry, let me clarify:
The search originally only looked at index and sourcetype; there was no hosts listed in the search. So anytime we saw events under that sourcetype it would alert.
Since then, the customer decided that they want to be able to search on all the hosts still, but only alert on higher value machines.
So, the original (working) search was:
index = abc_wineventlogs
sourcetype = abc_wineventlog_sec
I'm stumped. The changes that you made should not be causing the "per event" alerting at all. Is this a real-time alert?
Me too 🙂 This is a scheduled search (cron 5 minutes)
Could you try one thing? Create a new alert search altogether with this new search (with host) and see if that works. Disable the current one though to avoid duplicate alerts.