How to search for an event that doesn't have a response after 30 minutes?

[ramana4u]

I have two separate logs ( Request.log, and Response.log ).  

Events from App1 will be recorded in Request.log.

Events from App2 will be recorded in Response.log.  

Every request from App1 will receive a response from App2 within 30 minutes, and the response will be recorded in the Response.log file. 

App2 occasionally fails to reply within 30 minutes. Each event has a distinct field, which will be recorded in both log files.

How do I create an SPL query using these two distinct logs to search for the unsuccessful responses?

Any help?

[ramana4u]

Thanks for the response.   I have tried with stats and it was working I will explore the Transaction command as well.  

[gcusello]

Hi @ramana4u,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[ramana4u]

@gcusello 

request IDs are repeated in the logs.

How can we verify the response once each request is made? Using the request start time, we may check for a response within the next 30 minutes.

[gcusello · Accepted solution]

Hi @ramana4u,

the solution from @yuanliu will surely work, but I don't like transaction command because it's a very slow command And I prefer to use transaction command only if other solutions will not work.

So please try also this solution:

index=your_index source IN (Request.log, Response.log)
| eval kind=if(source="Request.log","Request","Response")
| stats 
   dc(kind) AS kind_count 
   values(kind) AS kind 
   earliest(eval(if( kind="Request",_time,""))) AS earliest 
   latest(eval(if( kind="Request",_time,""))) AS latest 
   BY requestID 
| eval duration=latest-earliest
| where (kind_count=1 AND kind="Request") OR (kind_count=2 AND duration>1800)

In addition: use always the index in your main search, you'll have faster searches.

Ciao.

Giuseppe

[yuanliu]

This is a classic use case for transaction.  If each requestID appears only once in each source, the following would do:

source IN (Request.log, Response.log)
| transaction maxspan=30m keeporphans=true requestID startswith=eval(source=="Request.log") endswith=eval(source=="Response.log")
| where closed_txn == "false"

Transaction is often expensive, so there is this guide Using stats instead of transaction.  Hope this helps.