How to use a lookup file to suppress alerts?

[mjshoaf]

We have a network load balancer (NLB) that generates syslog messages when servers fail to respond to health probes from the NLB. The message looks like this:

 Sep 1 20:20:22 HostA VirtualContextA: %ACE-3-251008: Health probe failed for server 10.10.10.10 on port 443...

I have a field extraction that pulls out the IP address and port number and creates the fields server_address and port. I would like to generate alerts (emails) when these messages are seen. However, I do not want the alerts to be generated if the servers are undergoing maintenance. I have a CSV file (inmaint.csv) that has a list of servers that are undergoing maintenance. The CSV file has the columns 'IP_address' and 'Caption' where 'Caption' is the server name.

My base search is: index=network "Health probe failed for server"

How do I add a lookup to this search to check the CSV file and determine whether a server is in maintenance?
Effectively, I want to suppress alerting for servers that are in the CSV file.

"If server_address is present in the lookup file, don't alert."

I'm quite confused about when to use lookup, inputlookup, subsearch, etc. I know WHAT I want to achieve, but can't figure out HOW. Please let me know if any of it's unclear.

[somesoni2 · Accepted solution]

Try something like this

Updated the extra punctuations

index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table  IP_address | rename IP_address as server_address ]

This will take all the IP_address present in the lookup, (rename it as server_address) and exclude them from your base search, so you'll not be alerted on the same.

[mjshoaf]

Fantastic! This achieves the goal. I just had to remove the odd punctuation so it looks like this:

 index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table  IP_address | rename IP_address as server_address]

Question: Does the initial '|' in the sub-search serve a purpose? It seems to work with or without it?

[somesoni2]

It does work with or without the initial pipe. The inputlookup on regular search needs a pipe, so I got the habit of putting that anyways.

[mjshoaf]

This one was simple conceptually, but I was really having a hard time with the syntax. Thank you very much for the assist with this!

[MuS]

Hi mjshoaf,

take a look at this answer http://answers.splunk.com/answers/289318/how-do-i-disable-monitoring-on-a-server-for-a-cert.html , it gives various methods to achieve what you want.

Hope this helps ...

cheers, MuS

[mjshoaf]

Yeah, this is the right idea, but my CSV is based on the results of a lookup to an external database. I can't control what columns are present in the CSV. My search simply needs to be able to check for the presence of server_address in the CSV.

If server_address is present in the CSV, don't alert. Otherwise, alert.

[MuS]

Try this :

your base search | search NOT [ | inputcsv yourcsvholdingserver_address.csv | table server_address ] | ...

This must be done in all your alerts or set it up as automatic lookup which will result in a new field (maybe call it alerts=disabled) and you can use it in your alerts like your base search NOT alert=disabled | ...

I hope this makes some sense ...

[mjshoaf]

Yes, thank you.

[diogofgm]

Assuming that both your data and the lookup have the same field name for the ip try this:

 index=network "Health probe failed for server" | append [|inputlookup my_lookupfile.csv ] | stats count by ip | where count = 1

If the field names are different just use |rename Command to rename one of the fields to be the same as the other.

[mjshoaf]

I don't understand what 'append' does. It seems like that would add results to my original results. I want to filter my original results and end up with a subset (i.e., just the ones that appy to servers that are not undergoing maintenance).