find request but not in response

[nmarun]

Hi,

[12:30:13 INF 0ceafa153290582e1f1faec3f98d84ac] Gateway API|Request...
[12:30:15 INF 0ceafa153290582e1f1faec3f98d84ac] Gateway API|Response...

These are sample request, response structures that we log. There are scenarios where a request might not have a response. I'd like to write a query to find such correlation id (highlighted).

This is something that I tried, but it's fetching all the matches.

index=pcf sourcetype="gateway-api" "Gateway API|Request"
| rex field=msg "INF ?(?P<correlationId>[a-zA-Z0-9-_, ]*)]"
| rex field=msg "INF ?(?P<correlationIdReq>[a-zA-Z0-9-_, ]*)]"
| table correlationId, correlationIdReq
| join type="left" correlationId
[search index=pcf sourcetype="gateway-api" "Gateway API|Response"
| rex field=msg "INF ?(?P<correlationId>[a-zA-Z0-9-_, ]*)]"
| rex field=msg "INF ?(?P<correlationIdRes>[a-zA-Z0-9-_, ]*)]"
| table correlationId, correlationIdReq, correlationIdRes]

Thanks,

Arun

[PickleRick]

First and foremost - don't use the "join" command unless you absolutely cannot avoid it.

In this case there's another command which you could use but it's also a "bad one" - it's "transaction".

But I supposed the "lightest" solution here would be to extract the correlationId as you did, extract the type of the call (Request/Response) and then do

| stats values(call) by correlationId

Then you can find only those which do not have responses

| where NOT call="Response"

(mind you that it's not the same as call!="Response")

Of course if you want to get some details of the requests, there will be more magic needed.

[nmarun]

@andrew_nelson, thanks for the quick reply sir. I'd like to extract a few other fields from the request and response (different fields in both, as highlighted below).

[12:30:13 INF 0ceafa153290582e1f1faec3f98d84ac] Gateway API|Request|GET|...
[12:30:15 INF 0ceafa153290582e1f1faec3f98d84ac] Gateway API|Response|200|ProductDetails|...

I'm not sure how I can make your query do that work and that's why I created the structure that I have. Please let me know how I can get all these done.

Thanks,

Arun

[andrew_nelson · Accepted solution]

You can write two custom extractions, one for each log type. Some rough extractions below. 
The request extraction:

| rex field=msg "Gateway API\|Request\|(?<method>[^\|]+)\|(?<field2>[^\|]+)\|(?<field3>[^\|]+)\|(?<field4>[^\|]+)\|(?<field5>[^\|]+)\|..."

The response extraction: 

| rex field=msg "Gateway API\|Response\|(?<response_code>\d+)\|(?<field2>[^\|]+)\|(?<field3>[^\|]+)\|(?<field4>[^\|]+)\|(?<field5>[^\|]+)\|..."

So assuming all of your fields are pipe separated ( | ) these extractions, will pull fields out for everything between the pipe symbol. You can rename the field2, field3 etc to what they actually are. 

If your dataset is actual event data, then its best to create those into proper field extractions that don't need to be created for every individual search.   

[nmarun]

Shoot.. I should've thought of that.

Thanks @andrew_nelson. 

[andrew_nelson]

Try something like this: 

index=pcf sourcetype="gateway-api" ("Gateway API|Request" OR "Gateway API|Response")
| rex field=msg "INF ?(?P<correlationId>[a-zA-Z0-9-_, ]*)]"
| rex field=msg "Gateway API\|(?<action_type>(Request|Response))"
| stats values(action_type) as action_type by correlationId 
| search action_type!="Response"