Splunk Enterprise Security: How to set up an alert when F5 LB is "down" for more than 15 minutes without an "up" message?

[wtaylor149]

I'm trying to setup a search to alert in ES when F5 LB is down for more than 15 minutes. The F5 LB only sends messages when there is a status change, i.e. the LB can't reach our vendor (MSSP).

index=f5 my_search_query_here | rex "-TCP member /Common/(?<my_vendor_host>\S+) monitor status (?<status>\w+)" | rex "\[ was (?<previous_state>[^\]]*)" | rex "mcpd\[\d+\]\:\s(?<message_code>\d+\:\d+)" | transaction my_vendor_host, status, message_code maxspan=45s maxevents=2 | rename host AS F5_Host | table _time F5_Host my_vendor_host status previous_state

Currently the search will pull both an up and down message. I'm looking to have it send an alert only when "down" message doesn't have an "up" message since the last time the search ran.

So, if the search runs every 15 minutes, search runs at top of the hour, finds a down message and based off of the host, does not find an up message in that same time frame, send an email. If that search finds a down, followed by an up message, don't send an email.

I hope I made sense. Thank you Splunk experts in advance.

[renjith_nair]

What about the below search

<your search> |stats latest(status) as status by host|search status="down" 

This should give you the latest status of the host and then search for down which gives you all hosts which don't have "up" status as latest status and then set an alert if no of events > 0

Hope this helps