topic Re: How to set up an alert if an ack message is not available for a particular req? in Splunk Search

How to set up an alert if an ack message is not available for a particular req?

prashanthberam — Mon, 13 Mar 2017 18:24:31 GMT

Hi,
i have messages like this how to setup an alert if ack message is not available in the logs for particular req.
and between req and rsp is more than 30 sec i need to setup an one more alert.

my logs like this:

2017-03-10 15:56:42.056 [WMQJCAResourceAdapter : 1] [INFO ] [DCN 0201706380692310C] SplunkLog - CorrelationID=000001806003698150190841, DCN=0201706380692310C, TransactionTimestamp=2017-03-10 15:56:37.742, GroupNumber =000Y69HB3, ServiceLinecount=4, SectionNumber=0008, CorporateEntityCode=OK1, ClaimType=0, VendorName=VERSCEND, VendorCode=CVP, TransactionCode=RSP, UtilizationAmount=3.75

2017-03-10 15:56:39.003 [WMQJCAResourceAdapter : 6] [INFO ] [DCN 0201706380692310C] SplunkLog - CorrelationID=000001806003698150190841, DCN=0201706380692310C, TransactionTimestamp=2017-03-10 15:56:39.002, GroupNumber =000Y69HB3, ServiceLinecount=4, SectionNumber=0008, CorporateEntityCode=OK1, ClaimType=0, VendorName=VERSCEND, VendorCode=CVP, TransactionCode=ACK, OutCome=C, Messagetext=ACCEPTED

2017-03-10 15:56:36.939 [WMQJCAResourceAdapter : 1] [INFO ] [DCN 0201706380692310C] SplunkLog - CorrelationID=000001806003698150190841, DCN=0201706380692310C, TransactionTimestamp=2017-03-10 15:56:36.939, GroupNumber =000Y69HB3, ServiceLinecount=4, SectionNumber=0008, CorporateEntityCode=OK1, ClaimType=0, VendorName=VERSCEND, VendorCode=CVP, TransactionCode=REQ

Re: How to set up an alert if an ack message is not available for a particular req?

woodcock — Mon, 13 Mar 2017 18:33:48 GMT

Like this:

Your Base Search Here | stats count list(_time) AS times range(_time) AS duration list(TransactionCode) AS TransactionCode BY  CorrelationID | search TransactionCode="REQ" AND NOT TransactionCode="ACK"

And this:

Your Base Search Here | stats count list(_time) AS times range(_time) AS duration list(TransactionCode) AS TransactionCode BY CorrelationID | search duration > 30 AND TransactionCode="REQ" AND TransactionCode="RSP"

Re: How to set up an alert if an ack message is not available for a particular req?

somesoni2 — Mon, 13 Mar 2017 18:36:01 GMT

Assuming there is a unique transaction ID available in log for each req-ack-rsp combination, you could do like this (assuming CorrelationID is the unique identifier, if there are multiple columns add them to stats's by clause)

Updated mv funtion

Alert when there is no ACK event for a transaction

your base search fetching all records
| stats min(_time) as StartTime  max(_time) as EndTime values(TransactionCode) as TransactionCodes by CorrelationID
| eval _time=StartTime | where isnull(mvfilter(match(TransactionCodes,"ACK")))

Alert when transaction duration is more than 30 sec

your base search fetching all records
| stats min(_time) as StartTime  max(_time) as EndTime values(TransactionCode) as TransactionCodes by CorrelationID
| eval _time=StartTime  | eval duration=EndTime-StartTime | where mvcount(TransactionCodes)=3 AND duration>30

Re: How to set up an alert if an ack message is not available for a particular req?

prashanthberam — Mon, 13 Mar 2017 18:54:07 GMT

while am searching first query am getting this Error in 'where' command: The arguments to the 'mvfind' function are invalid. what it means. may i know the reason.

Re: How to set up an alert if an ack message is not available for a particular req?

woodcock — Mon, 13 Mar 2017 18:55:55 GMT

I am done editing; sorry for the churn; I did not notice the 2nd part of the question.

Re: How to set up an alert if an ack message is not available for a particular req?

prashanthberam — Mon, 13 Mar 2017 18:59:31 GMT

ohh Np woodcock. thanks for helping.

Re: How to set up an alert if an ack message is not available for a particular req?

somesoni2 — Mon, 13 Mar 2017 19:03:05 GMT

Oops. Used wrong function. Just updated the query to use correct function.

Re: How to set up an alert if an ack message is not available for a particular req?

woodcock — Mon, 13 Mar 2017 19:06:23 GMT

I lied; I had an extra NOT in my 2nd answer. It is all good now.

Re: How to set up an alert if an ack message is not available for a particular req?

prashanthberam — Mon, 13 Mar 2017 19:13:01 GMT

Thanks for the help Somesoni2 . Now it's working ..

Re: How to set up an alert if an ack message is not available for a particular req?

prashanthberam — Mon, 13 Mar 2017 19:14:31 GMT

ya i haven't noticed that one. thanks woodcock