Get list of session id's that did not get a response from web service.

[phoeniix]

We are having a problem where requests are being sent to webservices but never return. I want to get a list of session numbers that sent a request to the webservice (XML Sent to Service) but did not get a response back (XML Response from Services). The following two lines are from Splunk that indexed our log4j logs of a successful purchase.

2013-07-26 14:48:45 - DEBUG:4LW5DNw1MdB5Rt5RhFUQCA__ - ExternalProcessManager.sendRequestToServices:(213) | XML Response from Services <?xml version="1.0" encoding="UTF-8"?>
<purchaseresponse haserror="0">
<price>39.02</price>
<product>G-200HA</product>
<actioncode>New</actioncode>

Show all 49 lines

host=CA1CW-WEB01 Options| sourcetype=log4j Options| source=ca1cw-web01\logs\log4j\csatp.2013-07-26.log Options

2013-07-26 14:48:45 - DEBUG:4LW5DNw1MdB5Rt5RhFUQCA__ - ExternalProcessManager.sendRequestToServices:(202) | XML Sent to Service http:/oursite.com/quote-rs/purchase: <?xml version="1.0" encoding="UTF-8"?>
<request><purchaserequest><actioncode>New</actioncode><aff>ha</aff><producer>HOMERTBO</producer><productclass>HOMEAWAYCP</productclass><plancode>A1</plancode>

Show all 49 lines

host=CA1CW-WEB01 Options| sourcetype=log4j Options| source=ca1cw-web01\logs\log4j\csatp.2013-07-26.log

How could we structure a query that uses rex to get the session number, then search for any session id's that did not log an (XML Response from Services)? I have tried this query and dont really know where to go from here:

sourcetype=log4j | rex ":\d\d - \S+:(?<sessionid>\S+) -.*&lt;request&gt;&lt;purchaserequest&gt;" | search &lt;response&gt;&lt;purchaseresponse&gt;

[lguinn2 · Accepted solution]

First, are you sure that your sessionid is extracting properly from both the request and response events?

Next, this may be closer to what you want...

sourcetype=log4j "purchaserequest" OR "purchaseresponse"
| rex "\:\d\d - S+:(?<sessionid>S+) -.*<purchaserequest>" 
| eval request=if(match(_raw,"\<purchaserequest\>",1,0)
| eval response=if(match(_raw,"\<purchaseresponse haserror=.*?\>",1,0)
| stats sum(request) as numRequests sum(response) as numReponses by sessionid
| where numRequests > numResponses

[phoeniix]

I am using this query to see by day what sessions had requests without responses:

sourcetype=log4j "XML Sent to Service" OR "XML Response from Services" 
| search "purchaserequest" OR "purchaseresponse"
| rex "(?\d\d\d\d\-\d\d\-\d\d) .* -\s+\S+:(?[^ ]+) -.*" 
| eval request=if(match(_raw,"\"),1,0)
| eval response=if(match(_raw,"\"),1,0)
| stats sum(request) as numRequests sum(response) as numResponses by date, sessionid
| where numRequests > numResponses
| sort date

[phoeniix]

Dang I love this product. There is always a way to do something.

You were right, I was missing a backslash in front of the S+ expression and that allowed logging of the session id. Also thanks for suggesting filtering out results before I process them, that speeds up the query quite a bit.

Then taking what you gave me and slightly tweaking it gave me some useful results.

[phoeniix]

I cannot update the original post, keep getting this field is required and all fields are filled in. The query should be this: sourcetype=log4j | rex "\:\d\d - \S+:(?\S+) -.*<purchaserequest>" | search <purchaseresponse>