How to check MQ reconnects after a connection is dropped

[bennch68]

Hi All,
I am rather hoping someone can assist me in creating a search that can be used for an alert to detect when a connection to MQ fails to re-connect for a system I am supporting.
I am new to Splunk and although I have found posts related to this topic, I have so far not been able to adapt them for my particular scenario.

I was hopeful the search below would suffice, but then realised it only works as I wanted it to if the MQ connection actually drops, otherwise the count evaluates as 0 and I end up with a false alerts.
 

index="sepa_instant"
source="D:\\Apps\\Instant_Sepa_01\\log\\ContinuityRequester*"
| transaction startswith="connection_down_error raised" maxspan=4m
| search "-INFOS- {3} QM reconnected"
| stats count
| where count="0"

Any assistance provided would be very much appreciated.

[VatsalJagani · Accepted solution]

@bennch68- You can use something like this:

index="sepa_instant"
source="D:\\Apps\\Instant_Sepa_01\\log\\ContinuityRequester*" ("connection_down_error raised" OR "-INFOS- {3} QM reconnected")
| sort 0 - _time
| eval event_type=if(match(_raw, "connection_down_error raised"), "disconnect", "connect")
| stats list(event_type) as event_type, list(_time) as all_timestamp values(eval(if(event_type="disconnect", _time, NULL()))) as time_of_disconnect
| search event_type="disconnect"
``` search part below may need to be reviewed and fixed, you can remove that part, see the results and update the query accordingly step-by-step ```
| eval index_of_disconnect=mvfind(event_type, "^disconnect$") | eval prev_disconnect=mvindex(event_type,index_of_disconnect-1) | eval next_disconnect=mvindex(event_type,index_of_disconnect+1)
| eval timestamp_disconnect=mvindex(all_timestamp, index_of_disconnect) | eval prev_timestamp=mvindex(all_timestamp,index_of_disconnect-1) | eval next_timestamp=mvindex(all_timestamp,index_of_disconnect+1) \
| eval prev_timediff=prev_timestamp-timestamp_disconnect, next_timediff=timestamp_disconnect-next_timestamp \
| search NOT (next_event_type="connect" next_timediff<500) | where timestamp_disconnect<=relative_time(now(),"-2m@m") AND timestamp_disconnect>=relative_time(now(),"-7m@m")

FYI, The search that you are trying to build isn't very simplest of search, so it may take some time to understand and learn. But it should give you many leanings along the way.

I hope this is helpful. Kindly upvote!!!

[bennch68]

The search appears to be working a treat.

Now just need to understand why, so lots to learn 😃

Thank you very much for your help.

Kind regards

Chris