Need to calculate difference between two epoch timestamps from two different searches where the correlation-ids are same

[Anurag_Ntt]

Hi Community,

I need to calculate the difference between two timestamps printed in log4j  logs of java application from two different searches, the timestamp is printed in the log after system time keyword in the logs. 

log for search -1

2024-07-18 06:11:23.438 INFO [ traceid=8d8f1bad8549e6ac6d1c864cbcb1f706 spanid=cdb1bb734ab9eedc ] com.filler.filler.filler.MessageLoggerVisitor [TLOG4-Thread-1-7] Jul 18,2024 06:11:23 GMT|91032|PRD|SYSTEM|test-01.Autodeploy-profiles-msgdeliver|10.12.163.65|-|-|-|-|com.filler.filler.filler.message.visitor.MessageLoggerVisitor|-|PRD01032 - Processor (Ingress Processor tlog-node4) processed message with system time 1721283083437 batch id d6e50727-ffe7-4db3-83a9-351e59148be2-23-0001 correlation-id (f00d9f9e-7534-4190-99ad-ffeea14859e5-23-0001) and body (

log for search-2

2024-07-18 06:11:23.487 INFO [ traceid= spanid= ]  com.filler.filler.filler.message.processor.RestPublisherProcessor [PRD-1] Jul 18,2024 06:11:23 GMT|91051|PRD|SYSTEM|test-01.Autodeploy-profiles-msgdeliver|10.12.163.65|-|-|-|-|com.filler.filler.filler.message.processor.RestPublisherProcessor|-|PRD01051 - Message with correlation-id f00d9f9e-7534-4190-99ad-ffeea14859e5-23-0001 successfully published at system time 1721283083487 to MCD

I am using below query to calculate the time difference but end up in duplicates and lot of null values, these null values are coming only when i do the calculations for individual searches null values don't pop up.

"sourcetype=log4j | rex "91032\|PRD\|SYSTEM\|test\-01\.Autodeploy\-profiles\-msgdeliver\|10\.12\.163\.65\|\-\|\-\|\-\|\-\|com\.filler\.filler\.filler\.message\.visitor\.MessageLoggerVisitor\|\-\|PRD01032 \- Processor (.*?) processed message with system time (?<systime_batch>.+) batch id (.*?) correlation-id \((?<corrid_batch>.+)\) and body" | rex "com\.filler\.filler.filler\.message\.processor\.RestPublisherProcessor\|\-\|PRD01051 \- Message with correlation\-id \((?<corrid_mcd>.+)\) successfully published at system time (?<systime_mcd>.+) to MCD" | dedup corrid_batch | eval diff = (systime_mcd-systime_batch) | where corrid_mcd=corrid_batch | table diff"

Kindly help in a

[ITWhisperer · Accepted solution]

You need to get the values into the same event so you can do the calculation - try something like this

sourcetype=log4j
| rex "91032\|PRD\|SYSTEM\|test\-01\.Autodeploy\-profiles\-msgdeliver\|10\.12\.163\.65\|\-\|\-\|\-\|\-\|com\.filler\.filler\.filler\.message\.visitor\.MessageLoggerVisitor\|\-\|PRD01032 \- Processor (.*?) processed message with system time (?<systime_batch>.+) batch id (.*?) correlation-id \((?<corrid>.+)\) and body"
| rex "com\.filler\.filler.filler\.message\.processor\.RestPublisherProcessor\|\-\|PRD01051 \- Message with correlation\-id \((?<corrid>.+)\) successfully published at system time (?<systime_mcd>.+) to MCD" 
| stats first(systime_batch) as systime_batch values(systime_mcd) as systime_mcd by corrid
| eval diff = (systime_mcd-systime_batch)

[Anurag_Ntt]

Awesome.. Thanks @ITWhisperer  worked like a charm 🙂