Splunk Search

How to use a lookup file to suppress alerts?

mjshoaf
New Member

We have a network load balancer (NLB) that generates syslog messages when servers fail to respond to health probes from the NLB. The message looks like this:

 Sep 1 20:20:22 HostA VirtualContextA: %ACE-3-251008: Health probe failed for server 10.10.10.10 on port 443...

I have a field extraction that pulls out the IP address and port number and creates the fields server_address and port. I would like to generate alerts (emails) when these messages are seen. However, I do not want the alerts to be generated if the servers are undergoing maintenance. I have a CSV file (inmaint.csv) that has a list of servers that are undergoing maintenance. The CSV file has the columns 'IP_address' and 'Caption' where 'Caption' is the server name.

My base search is: index=network "Health probe failed for server"

How do I add a lookup to this search to check the CSV file and determine whether a server is in maintenance?
Effectively, I want to suppress alerting for servers that are in the CSV file.

"If server_address is present in the lookup file, don't alert."

I'm quite confused about when to use lookup, inputlookup, subsearch, etc. I know WHAT I want to achieve, but can't figure out HOW. Please let me know if any of it's unclear.

Tags (2)
0 Karma
1 Solution

somesoni2
Revered Legend

Try something like this

Updated the extra punctuations

index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table  IP_address | rename IP_address as server_address ]

This will take all the IP_address present in the lookup, (rename it as server_address) and exclude them from your base search, so you'll not be alerted on the same.

View solution in original post

somesoni2
Revered Legend

Try something like this

Updated the extra punctuations

index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table  IP_address | rename IP_address as server_address ]

This will take all the IP_address present in the lookup, (rename it as server_address) and exclude them from your base search, so you'll not be alerted on the same.

View solution in original post

mjshoaf
New Member

Fantastic! This achieves the goal. I just had to remove the odd punctuation so it looks like this:

 index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table  IP_address | rename IP_address as server_address]

Question: Does the initial '|' in the sub-search serve a purpose? It seems to work with or without it?

0 Karma

somesoni2
Revered Legend

It does work with or without the initial pipe. The inputlookup on regular search needs a pipe, so I got the habit of putting that anyways.

0 Karma

mjshoaf
New Member

This one was simple conceptually, but I was really having a hard time with the syntax. Thank you very much for the assist with this!

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi mjshoaf,

take a look at this answer http://answers.splunk.com/answers/289318/how-do-i-disable-monitoring-on-a-server-for-a-cert.html , it gives various methods to achieve what you want.

Hope this helps ...

cheers, MuS

0 Karma

mjshoaf
New Member

Yeah, this is the right idea, but my CSV is based on the results of a lookup to an external database. I can't control what columns are present in the CSV. My search simply needs to be able to check for the presence of server_address in the CSV.

If server_address is present in the CSV, don't alert. Otherwise, alert.

0 Karma

MuS
SplunkTrust
SplunkTrust

Try this :

your base search | search NOT [ | inputcsv yourcsvholdingserver_address.csv | table server_address ] | ...

This must be done in all your alerts or set it up as automatic lookup which will result in a new field (maybe call it alerts=disabled) and you can use it in your alerts like your base search NOT alert=disabled | ...

I hope this makes some sense ...

mjshoaf
New Member

Yes, thank you.

0 Karma

diogofgm
SplunkTrust
SplunkTrust

Assuming that both your data and the lookup have the same field name for the ip try this:

 index=network "Health probe failed for server" | append [|inputlookup my_lookupfile.csv ] | stats count by ip | where count = 1

If the field names are different just use |rename Command to rename one of the fields to be the same as the other.

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

mjshoaf
New Member

I don't understand what 'append' does. It seems like that would add results to my original results. I want to filter my original results and end up with a subset (i.e., just the ones that appy to servers that are not undergoing maintenance).

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!