I'm trying to setup a search to alert in ES when F5 LB is down for more than 15 minutes. The F5 LB only sends messages when there is a status change, i.e. the LB can't reach our vendor (MSSP).
index=f5 my_search_query_here | rex "-TCP member /Common/(?<my_vendor_host>\S+) monitor status (?<status>\w+)" | rex "\[ was (?<previous_state>[^\]]*)" | rex "mcpd\[\d+\]\:\s(?<message_code>\d+\:\d+)" | transaction my_vendor_host, status, message_code maxspan=45s maxevents=2 | rename host AS F5_Host | table _time F5_Host my_vendor_host status previous_state
Currently the search will pull both an up and down message. I'm looking to have it send an alert only when "down" message doesn't have an "up" message since the last time the search ran.
So, if the search runs every 15 minutes, search runs at top of the hour, finds a down message and based off of the host, does not find an up message in that same time frame, send an email. If that search finds a down, followed by an up message, don't send an email.
I hope I made sense. Thank you Splunk experts in advance.