topic Re: Calculate time difference in two different logs in Splunk Search

Calculate time difference in two different logs

cj039165 — Tue, 19 Jul 2016 14:34:26 GMT

Hello -

Stumped on this. I have two different log files. One logs the time (and data) in transactions sent, the other has the time (and data) received. I would like to calculate the 'response' time. From there we could could alert if it goes above a set period of time. Each transactions is link between the two logs via a thread id. I've very new to Splunk, any help would be great.

Log example Send:
10:31:55,462 DEBUG [Thread-216189]

Log example Receive:
10:32:03,067 DEBUG [Thread-216189]

Re: Calculate time difference in two different logs

somesoni2 — Tue, 19 Jul 2016 15:48:08 GMT

Try like this (check the field name)
Updated

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

If above still doesn't work, try this

(base search for log 1 - transaction sent ) OR (base search for log 2 - transaction received) | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

Re: Calculate time difference in two different logs

cj039165 — Tue, 19 Jul 2016 16:26:47 GMT

Hello. Thanks for the response. I ran the following search. I'm getting the following error: Error in 'eval' command: Typechecking failed. '-' only takes numbers.

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats values(_time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=mvindex(Time,1)-mvindex(Time,0) | convert ctime(Time)

Re: Calculate time difference in two different logs

sundareshr — Tue, 19 Jul 2016 16:44:51 GMT

Try transaction

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | transaction thread_id | rename duration AS response_time | table thread_id response_time

Re: Calculate time difference in two different logs

cj039165 — Tue, 19 Jul 2016 16:53:52 GMT

Ran the search. I don't get an error, however I don't see any search results. Do I need to extract [thread-345432] into a extracted field?

Thanks

Re: Calculate time difference in two different logs

somesoni2 — Tue, 19 Jul 2016 17:00:19 GMT

Try the updated answer.

Re: Calculate time difference in two different logs

sundareshr — Tue, 19 Jul 2016 17:02:23 GMT

You will have to extract the thread_id field. Try this (you may need to correct sent & received as well)

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | rex "Thread-(?<thread_id>\d+)" | transaction thread_id startswith="sent" endswith="received" | rename duration AS response_time | table thread_id response_time

Re: Calculate time difference in two different logs

cj039165 — Tue, 19 Jul 2016 17:27:20 GMT

I did your second search after extracting the thread-id. I'm see what I needed. Thanks very much for the help!

Re: Calculate time difference in two different logs

apilger_splunk — Tue, 19 Jul 2016 17:45:45 GMT

Try this:

source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/KAPE_Send.log" | stats min(_time) AS my_start,max(_time) AS my_end by  thread_id| eval reponse_time=my_end-my_start

Re: Calculate time difference in two different logs

cj039165 — Tue, 29 Sep 2020 10:23:49 GMT

Noticed a problem with the search results. It turns out it's possible to have the same thread-id in on multiple servers. For example [Thread-216189] can be in UNITED_Send.log / UNITED_Receive.log on HOST WASAPP07 or WASAPP08 or WASAPP09. How do I get it to not calculate across the hosts?

Search String:

index=hdx_payer source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Receive.log" OR source="/hdx2/was70-32/AppServer/profiles/AppSrv01/logs/PRD2_PY0/UNITED_Send.log" | eval Time=_time | stats values(Time) as Time by thread_id | where mvcount(Time)=2 | eval response_time=tonumber(mvindex(Time,1))-tonumber(mvindex(Time,0)) | convert ctime(Time)

Re: Calculate time difference in two different logs

somesoni2 — Tue, 26 Jul 2016 19:36:54 GMT

Just include the host field in the transaction command

Re: Calculate time difference in two different logs

cj039165 — Tue, 26 Jul 2016 20:39:48 GMT

Tried it yesterday. For some reason that did not work. It appears adding host=* shows the right data. Will review with the group looking for the info. Thanks.