Calculate time difference in two different logs

[cj039165]

Hello -

Stumped on this. I have two different log files. One logs the time (and data) in transactions sent, the other has the time (and data) received. I would like to calculate the 'response' time. From there we could could alert if it goes above a set period of time. Each transactions is link between the two logs via a thread id. I've very new to Splunk, any help would be great.

Log example Send:
10:31:55,462 DEBUG [Thread-216189]

Log example Receive:
10:32:03,067 DEBUG [Thread-216189]

[cj039165]

Noticed a problem with the search results. It turns out it's possible to have the same thread-id in on multiple servers. For example [Thread-216189] can be in UNITED_Send.log / UNITED_Receive.log on HOST WASAPP07 or WASAPP08 or WASAPP09. How do I get it to not calculate across the hosts?

Search String:

index=hdx_payer source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Send.log" | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

[apilger_splunk]

Try this:

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats min(_time) AS my_start,max(_time) AS my_end by  thread_id| eval reponse_time=my_end-my_start

[sundareshr]

Try transaction

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | transaction thread_id | rename duration AS response_time | table thread_id response_time 

[cj039165]

Ran the search. I don't get an error, however I don't see any search results. Do I need to extract [thread-345432] into a extracted field?

Thanks

[sundareshr]

You will have to extract the thread_id field. Try this (you may need to correct sent & received as well)

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | rex "Thread-(?<thread_id>\d+)" | transaction thread_id startswith="sent" endswith="received" | rename duration AS response_time | table thread_id response_time 

[somesoni2]

Just include the host field in the transaction command

[cj039165]

Tried it yesterday. For some reason that did not work. It appears adding host=* shows the right data. Will review with the group looking for the info. Thanks.

[somesoni2]

Try like this (check the field name)
Updated

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

If above still doesn't work, try this

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

[cj039165]

Hello. Thanks for the response. I ran the following search. I'm getting the following error: Error in 'eval' command: Typechecking failed. '-' only takes numbers.

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=mvindex(Time,1)-mvindex(Time,0) | convert ctime(Time)

[somesoni2]

Try the updated answer.

[cj039165]

I did your second search after extracting the thread-id. I'm see what I needed. Thanks very much for the help!