Splunk Search

Inhibiting alerts from saved searches that had search errors

teedilo
Path Finder

Is there a way to inhibit alerts from saved searches that had errors? Saved searches will sometimes fail with errors like the one below:

-- Search generated the following messages --
Message Level: WARN
1. Unable to distribute to peer named at uri https://:11004 because replication was unsuccessful. replicationStatus Failed

We are aware of the causes for this and other fairly common errors that cause searches to fail or deliver incomplete search results. This is typically tolerable if the search failures are fairly infrequent, especially for saved searches where partial search results are sometimes good enough. However, some of our saved searches do not tolerate search errors. For example, we have a saved search that is configured to alert for the condition "if number of hosts is less than N", where N is the number of hosts from which we expect to get search results. Search errors will result in numbers less than N, causing alerts for this saved search.

Any suggestions for handling search errors for saved searches? I'm thinking that a scripted alert solution may be required, though I haven't found any documentation that indicates how a script would check for any search errors that might occur. Any suggestions or examples would be appreciated. Thanks.

Tags (1)
1 Solution

MuS
SplunkTrust
SplunkTrust

Okay, here is how you can do this. But be warned; you can easily break stuff here and also keep in mind that the default $SPLUNK_HOME/etc/apps/search/bin/sendemail.py can be over-written with any Splunk update.

So what must be done, to filter out alerts fired because a search failed on some search peer?

Backup the Splunk provided $SPLUNK_HOME/etc/apps/search/bin/sendemail.py and open in your favorite editor.

Go to line number 249 which contains the following:

body.write(intro)

this is the command to write the email content. Now we add the following lines JUST BEFORE line 249. What out for Python indentation, again: you can break stuff here!

# filter failed searches because of ..... foo bar
myList = ['Search generated the following messages', 'Search results may be incomplete']
for myError in myList:
 if myError in intro:
    admin = "SomeAdminEmail@spam.com"
    recipients = []
    message.replace_header('To', admin)
    recipients.extend(EMAIL_DELIM.split(admin))
    subject="Search failed, get up and check why...."
    message['Subject'] = subject

Above lines will check intro for any value set in myList. If there is a match the email reciepient To will be changed to be the admin email set in admin. Also it changes the alert subject to what ever is set in subject above.

That's it, if you done everything correct only the defined admin will recieve those alerts.

Again, you've been warned and don't blame me if you break something.....

hope this helps ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Okay, here is how you can do this. But be warned; you can easily break stuff here and also keep in mind that the default $SPLUNK_HOME/etc/apps/search/bin/sendemail.py can be over-written with any Splunk update.

So what must be done, to filter out alerts fired because a search failed on some search peer?

Backup the Splunk provided $SPLUNK_HOME/etc/apps/search/bin/sendemail.py and open in your favorite editor.

Go to line number 249 which contains the following:

body.write(intro)

this is the command to write the email content. Now we add the following lines JUST BEFORE line 249. What out for Python indentation, again: you can break stuff here!

# filter failed searches because of ..... foo bar
myList = ['Search generated the following messages', 'Search results may be incomplete']
for myError in myList:
 if myError in intro:
    admin = "SomeAdminEmail@spam.com"
    recipients = []
    message.replace_header('To', admin)
    recipients.extend(EMAIL_DELIM.split(admin))
    subject="Search failed, get up and check why...."
    message['Subject'] = subject

Above lines will check intro for any value set in myList. If there is a match the email reciepient To will be changed to be the admin email set in admin. Also it changes the alert subject to what ever is set in subject above.

That's it, if you done everything correct only the defined admin will recieve those alerts.

Again, you've been warned and don't blame me if you break something.....

hope this helps ...

cheers, MuS

linu1988
Champion

Thank you Mus, good solution. And if you are concerned about the sendmail.py file , we can use this in another app which uses it's own sendmail.py version. Hence less chance of being overwritten. 🙂

0 Karma

MuS
SplunkTrust
SplunkTrust

I can confirm that the script works fine now. Allow some time write down everything and as soon as it is ready I will post it here....

0 Karma

linu1988
Champion

@Mus, Thank you for your attention and efforts 🙂

0 Karma

MuS
SplunkTrust
SplunkTrust

@linu1988 before I'll do this I have to verify the script still works and does what it should. If all is verified I will for sure post it here and file an enhancement request to add it to the original sendemail.py shipped with Splunk.

linu1988
Champion

Mus could you post the same in the answer section. I know this in nothing but error generated by splunk which is handled and send in the email content.

Sendmail.py has a block where it catches the error. But we all would be grateful if you could just post your idea and guide us towards the solution. Splunk provided so many things,but it seems many many things the user has to figure out themselves rather any provided splunk functionality.

0 Karma

MuS
SplunkTrust
SplunkTrust

found a solution! working on the alert script.....

0 Karma

MuS
SplunkTrust
SplunkTrust

Looks like I found a way to achieve this.....but I have to verify this first....

0 Karma

jtrucks
Splunk Employee
Splunk Employee

I would love to see an option to abort an alert if there is an error condition encountered during the search. Perhaps with the added bonus of firing an alert to designated (or all?) admins to investigate the error(s).

--
Jesse Trucks
Minister of Magic

MuS
SplunkTrust
SplunkTrust

Did you made any progress on this topic?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...