We have a network load balancer (NLB) that generates syslog messages when servers fail to respond to health probes from the NLB. The message looks like this:
Sep 1 20:20:22 HostA VirtualContextA: %ACE-3-251008: Health probe failed for server 10.10.10.10 on port 443...
I have a field extraction that pulls out the IP address and port number and creates the fields server_address and port. I would like to generate alerts (emails) when these messages are seen. However, I do not want the alerts to be generated if the servers are undergoing maintenance. I have a CSV file (inmaint.csv) that has a list of servers that are undergoing maintenance. The CSV file has the columns 'IP_address' and 'Caption' where 'Caption' is the server name.
My base search is: index=network "Health probe failed for server"
How do I add a lookup to this search to check the CSV file and determine whether a server is in maintenance?
Effectively, I want to suppress alerting for servers that are in the CSV file.
"If server_address is present in the lookup file, don't alert."
I'm quite confused about when to use lookup, inputlookup, subsearch, etc. I know WHAT I want to achieve, but can't figure out HOW. Please let me know if any of it's unclear.
Try something like this
Updated the extra punctuations
index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table IP_address | rename IP_address as server_address ]
This will take all the IP_address present in the lookup, (rename it as server_address) and exclude them from your base search, so you'll not be alerted on the same.
Try something like this
Updated the extra punctuations
index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table IP_address | rename IP_address as server_address ]
This will take all the IP_address present in the lookup, (rename it as server_address) and exclude them from your base search, so you'll not be alerted on the same.
Fantastic! This achieves the goal. I just had to remove the odd punctuation so it looks like this:
index=network "Health probe failed for server" NOT [|inputlookup inmaint.csv | table IP_address | rename IP_address as server_address]
Question: Does the initial '|' in the sub-search serve a purpose? It seems to work with or without it?
It does work with or without the initial pipe. The inputlookup on regular search needs a pipe, so I got the habit of putting that anyways.
This one was simple conceptually, but I was really having a hard time with the syntax. Thank you very much for the assist with this!
Hi mjshoaf,
take a look at this answer http://answers.splunk.com/answers/289318/how-do-i-disable-monitoring-on-a-server-for-a-cert.html , it gives various methods to achieve what you want.
Hope this helps ...
cheers, MuS
Yeah, this is the right idea, but my CSV is based on the results of a lookup to an external database. I can't control what columns are present in the CSV. My search simply needs to be able to check for the presence of server_address in the CSV.
If server_address is present in the CSV, don't alert. Otherwise, alert.
Try this :
your base search | search NOT [ | inputcsv yourcsvholdingserver_address.csv | table server_address ] | ...
This must be done in all your alerts or set it up as automatic lookup which will result in a new field (maybe call it alerts=disabled
) and you can use it in your alerts like your base search NOT alert=disabled | ...
I hope this makes some sense ...
Yes, thank you.
Assuming that both your data and the lookup have the same field name for the ip try this:
index=network "Health probe failed for server" | append [|inputlookup my_lookupfile.csv ] | stats count by ip | where count = 1
If the field names are different just use |rename
Command to rename one of the fields to be the same as the other.
I don't understand what 'append' does. It seems like that would add results to my original results. I want to filter my original results and end up with a subset (i.e., just the ones that appy to servers that are not undergoing maintenance).