Splunk Search

Calculate time difference in two different logs

cj039165
New Member

Hello -

Stumped on this. I have two different log files. One logs the time (and data) in transactions sent, the other has the time (and data) received. I would like to calculate the 'response' time. From there we could could alert if it goes above a set period of time. Each transactions is link between the two logs via a thread id. I've very new to Splunk, any help would be great.

Log example Send:
10:31:55,462 DEBUG [Thread-216189]

Log example Receive:
10:32:03,067 DEBUG [Thread-216189]

0 Karma

cj039165
New Member

Noticed a problem with the search results. It turns out it's possible to have the same thread-id in on multiple servers. For example [Thread-216189] can be in UNITED_Send.log / UNITED_Receive.log on HOST WASAPP07 or WASAPP08 or WASAPP09. How do I get it to not calculate across the hosts?

Search String:

index=hdx_payer source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Send.log" | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)
0 Karma

apilger_splunk
Splunk Employee
Splunk Employee

Try this:

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats min(_time) AS my_start,max(_time) AS my_end by  thread_id| eval reponse_time=my_end-my_start
0 Karma

sundareshr
Legend

Try transaction

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | transaction thread_id | rename duration AS response_time | table thread_id response_time 
0 Karma

cj039165
New Member

Ran the search. I don't get an error, however I don't see any search results. Do I need to extract [thread-345432] into a extracted field?

Thanks

0 Karma

sundareshr
Legend

You will have to extract the thread_id field. Try this (you may need to correct sent & received as well)

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | rex "Thread-(?<thread_id>\d+)" | transaction thread_id startswith="sent" endswith="received" | rename duration AS response_time | table thread_id response_time 
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Just include the host field in the transaction command

0 Karma

cj039165
New Member

Tried it yesterday. For some reason that did not work. It appears adding host=* shows the right data. Will review with the group looking for the info. Thanks.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try like this (check the field name)
Updated

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

If above still doesn't work, try this

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)
0 Karma

cj039165
New Member

Hello. Thanks for the response. I ran the following search. I'm getting the following error: Error in 'eval' command: Typechecking failed. '-' only takes numbers.

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=mvindex(Time,1)-mvindex(Time,0) | convert ctime(Time)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try the updated answer.

0 Karma

cj039165
New Member

I did your second search after extracting the thread-id. I'm see what I needed. Thanks very much for the help!

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...