Need to calculate difference between two epoch timestamps from different searches

[Anurag_Ntt]

Hi Community, I need to calculate the difference between two timestamps printed in log4j logs of java application from 3 different searches, the timestamp is printed in the log after system time keyword in the logs.

Logs for search -1

2024-07-18 06:11:23.438 INFO [ traceid=8d8f1bad8549e6ac6d1c864cbcb1f706 spanid=cdb1bb734ab9eedc ] com.filler.filler.filler.MessageLoggerVisitor [TLOG4-Thread-1-7] Jul 18,2024 06:11:23 GMT|91032|PRD|SYSTEM|test-01.Autodeploy-profiles-msgdeliver|10.12.163.65|-|-|-|-|com.filler.filler.filler.message.visitor.MessageLoggerVisitor|-|PRD01032 - Processor (Ingress Processor tlog-node4) processed message with system time 1721283083437 batch id d6e50727-ffe7-4db3-83a9-351e59148be2-23-0001 correlation-id (f00d9f9e-7534-4190-99ad-ffeea14859e5-23-0001) and body (

Logs for search -2

DFM01081 - Batch having id d6e50727-ffe7-4db3-83a9-351e59148be2-23-0001 on processor-name Egress Processor, transaction status commited by consumer

Logs for search-3

2024-07-18 06:11:23.487 INFO [ traceid= spanid= ] com.filler.filler.filler.message.processor.RestPublisherProcessor [PRD-1] Jul 18,2024 06:11:23 GMT|91051|PRD|SYSTEM|test-01.Autodeploy-profiles-msgdeliver|10.12.163.65|-|-|-|-|com.filler.filler.filler.message.processor.RestPublisherProcessor|-|PRD01051 - Message with correlation-id f00d9f9e-7534-4190-99ad-ffeea14859e5-23-0001 successfully published at system time 1721283083487 to MCD

I am using below query to calculate the time difference. I need to filter out the correlation ids in search 1not matching the batch ids from search 2 and calculate the systime difference from the matching correlation ids b/w search-1 and search-2 which also match with search-3. The below query gives empty systime_mcd need help in getting this through

sourcetype=log4j | rex "91032\|PRD\|SYSTEM\|test\-01\.Autodeploy\-profiles\-msgdeliver\|10\.12\.163\.65\|\-\|\-\|\-\|\-\|com\.filler\.filler\.filler\.message\.visitor\.MessageLoggerVisitor\|\-\|PRD01032 \- Processor (.*?) processed message with system time (?.+) batch id (?.+) correlation-id \((?.+)\) and body" |rex "DFM01081 \- Batch having id (?.+) on processor-name Egress Processor\, transaction status commited by consumer | rex "com\.filler\.filler.filler\.message\.processor\.RestPublisherProcessor\|\-\|PRD01051 \- Message with correlation\-id \((?.+)\) successfully published at system time (?.+) to MCD" | stats first(systime_batch) as systime_batch values(systime_mcd) as systime_mcd values(corrid) as corrid by batch_id_passed | mvexpand corrid | eval diff = (systime_mcd-systime_batch)

@ITWhisperer   can you please look into this as well, this is an extension of what you already helped with.

Thanks in advance

[PickleRick]

1. Please don't call out specific people. It's rude. If you demand someone's help you typically pay for consulting services. Here people help in own spare time out of good will.

2. When you post samples and SPL excerpts, please format them properly - in code block or preformatted paragraphs (and use line breaking for SPL)

3.  Did you verify that before you do the stats the  fields you're aggregating are properly extracted?

4. stats values() can produce multivalued fields - trying to treat them as simple integers won't work

5. As you're extracting fields from textual content, you might need to call tonumber() on them to get an integer which you can use to calculate difference.

[Anurag_Ntt]

1- Point taken, was not a demand but a request not sure how could  have I framed it to look like a request 🙂 will avoid tagging people

2-Did that thanks for the feedback

3/4/5- Data is getting extracted properly but for the systime_mcd  which is null for all the correlation-ids.

[PickleRick]

Your rex commands do not seem to contain any named capture groups so how are your files extracted?