Splunk Search

How to edit my search to alert once per result for multiple hosts?

CaptainHook
Communicator

We are using Splunk 6.4.2 and I have alerting setup on a specific search as follows:

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN

We would like to be alerted for each event that comes up on the individual host; however, when the alerting happens, it creates multiple events under one alert. Unfortunately, due to the request of only wanting to alert on the specific 10 hosts out of 25 hosts, I have to include the host names in the search. I do have alert mode set to: "Once per Result". Is there something that can be changed so Splunk alerts if any of these hosts events show? As stated, we would like it to be one alert for each event.

Should I change the search or do I need to set up alerting individually for each one? Any suggestions would be greatly appreciated.

Thank you.

1 Solution

CaptainHook
Communicator

I was able to get the customer online to test and none of the changes we made affected the alerting to fire one per event. I went ahead and re-created the alerting with a fresh copy and we are getting better results now. Seems we just had a corrupt alert.

Thank you for all the suggestions and assistance.

View solution in original post

0 Karma

CaptainHook
Communicator

I was able to get the customer online to test and none of the changes we made affected the alerting to fire one per event. I went ahead and re-created the alerting with a fresh copy and we are getting better results now. Seems we just had a corrupt alert.

Thank you for all the suggestions and assistance.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I converted this to an answer so you can mark this one as accepted. https://www.youtube.com/watch?v=h-wfO6_eq-A

0 Karma

CaptainHook
Communicator

I appreciate the added self-help link. 🙂

0 Karma

sloshburch
Splunk Employee
Splunk Employee

This sounds like a classic use of the Alert mode features. Perhaps you have it set up correctly but didn't put the Host in the throttling fields? You'll want to put the unique item in that field so it will alert once per result but then ignore (throttle) if it sees the same one again within the time frame you listed.

Also, what is the full search string? For sanity, we should make sure your search string isn't undermining your alerting.

alt text

CaptainHook
Communicator

Thanks Burch. Yes, we do not want to throttle as they need to see all events (logins) from each host.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

It sounds like perhaps the customer is responsible for a subset of the hosts that are contained in the result set. For example, the search may produce alerts for hosts host1, host2, host 3, and host4 but this customer wants one alert when either host1 or host3 trigger.

If that's accurate, then you can create a flag (if you will) for their hosts and then use that flag as the suppression:

eval flag = if( host=="host1" , "annoying_customer", "pleasant_customer" )

Make sense or did I still not understand?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Assuming the search is the exact alert search, try something like this. basically combining the search to merge all events related to a host as one row, so you get one alert per host.

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | transaction host

OR

index = wineventlogs 
sourcetype = wineventlog_sec 
host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | stats values(_raw) as _raw by host
0 Karma

CaptainHook
Communicator

Thank you, I will try adding that to my search now and will update the post accordingly. I appreciate your expertise and assistance.

0 Karma

CaptainHook
Communicator

Unfortunately, neither of these have worked. Any other suggestions or even different ways to monitor the events individually?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this as well

index = wineventlogs 
 sourcetype = wineventlog_sec 
 host=SERVER12VWIN OR host=SERVER82VWIN OR host=SERVER13VWIN OR host=SERVER81VWIN OR host=SERVER180VWN OR host=SERVER14VWIN OR host=SERVER15VWIN OR host=SERVER83VWIN OR host=SERVER001VWIN OR host=SERVER049VWIN | dedup host

Also, when you say it didn't work, what was happening?

0 Karma

CaptainHook
Communicator

I will try that now.

The alerting works, but it will group all the hosts into one alert. So, the file (pdf) that gets emailed to our support teams will contain multiple events, instead of one pdf (alert) for each host.

The goal is to have one host alert and then a separate ticket is automatically generated to the support teams. Currently, it is opening one ticket for multiple hosts.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Is the alert properly configured to send email per event? When you edit the alert from Alerts dashboard, in the 2nd page, does it say "per event" under Action options section?

0 Karma

CaptainHook
Communicator

It is, yes. It worked prior when I did not include hosts in the search. However, the customer would like to only report on a number of hosts and not all.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So before you added host name in the search, you're getting 25 emails, one for each host? I'm assuming that's what you want now, 10 emails, one for each host?

0 Karma

CaptainHook
Communicator

That is correct. Now, we only want to alert on 10 of the 25 hosts. One email for each host as it occurs. In practice, we want to be alerted whenever someone logs into the host via a specific method.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

If the alert was sending one email per host earlier, it should be doing the same, after explicitly selecting fewer hosts. Has anything else changed in the alert? Is the query that you posted the full query? If not, then could you post full alert-search what was before and what you changed into?

0 Karma

CaptainHook
Communicator

Sorry, let me clarify:
The search originally only looked at index and sourcetype; there was no hosts listed in the search. So anytime we saw events under that sourcetype it would alert.

Since then, the customer decided that they want to be able to search on all the hosts still, but only alert on higher value machines.

So, the original (working) search was:
index = abc_wineventlogs
sourcetype = abc_wineventlog_sec

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I'm stumped. The changes that you made should not be causing the "per event" alerting at all. Is this a real-time alert?

0 Karma

CaptainHook
Communicator

Me too 🙂 This is a scheduled search (cron 5 minutes)

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Could you try one thing? Create a new alert search altogether with this new search (with host) and see if that works. Disable the current one though to avoid duplicate alerts.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...