topic Re: timestamp difference between two different events with matching criteria in Splunk Dev

timestamp difference between two different events with matching criteria

shaganga — Tue, 29 Sep 2020 15:02:37 GMT

All, need helps... Please find the mappings and portion of logs

timeStamp=2017/07/20 01:43:06.78
queueName=GFSPII.QUEUE
MsgId=414D512044455647505030332020202059635ADE2017BA02
CoreId=414D512044455647505030332020202059635ADE2017BA02
MsgDefId=p001
transactionId=BLANK
ClearSysRef=BLANK
MsgId=NATAAU33XXXN2017720591500514979309
OriginalMsgId=BLANK
InstructionId=NATAAU33XXXN2017720591500514979309
OriginalInstructionId=BLANK

the requirements is below
A. find the record for (queue=GFSPII.QUEUE and MsgDefId=p001 event) and retrieve the timeStamp(01:43:06.78), InstructionId(NATAAU33XXXN2017720591500514979309) for this event
B. Find the record for the queue=GTAPIO.QUEUE and MsgDefId=p002 with the InstructionId=NATAAU33XXXN2017720591500514979309 (which u retrieve in Step A) should match with OriginalInstructionId=NATAAU33XXXN2017720591500514979309 in STEP B and retrieve the timeStamp(01:43:53.70) value for this event (in this example this queue occurs fifth or sixth event in log)
C. provide the difference of timeStamp values which we got in A and B

Is it possible?
1. we have 1000+ queues in the scenarios, where single transaction flow contains five or six events or more
2. we need to calculate how many transactions which are exceed ( difference between timestamps or > 1.2 seconds)

2017/07/20 01:43:06.78,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017BA02,414D512044455647505030332020202059635ADE2017BA02,p001,,,NATAAU33XXXN2017720591500514979309,,NATAAU33XXXN2017720591500514979309,
2017/07/20 01:43:18.91,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DF1,414D512044455647505030332020202059635ADE10044DF1,pacs.008.001.02,,H07KB4306M850R1X,H07KB4315GZ50R21GPPS,,NATAAU33XXXN2017720591500514979309,
2017/07/20 01:43:30.22,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017BC02,414D512044455647505030332020202059635ADE2017BC02,p001,,,NATAAU33XXXN201772001500514980144,,NATAAU33XXXN201772001500514980144,
2017/07/20 01:43:30.50,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DF2,414D512044455647505030332020202059635ADE10044DF2,pacs.008.001.02,,H07KB43306950R25,H07KB4330CM50R29GPPS,,NATAAU33XXXN201772001500514980144,
2017/07/20 01:43:50.92,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017BC05,414D512044455647505030332020202059635ADE2017BC05,pacs.002.001.03,,H07KB43306950R25,1230123789422017720481500515028412,H07KB4330CM50R29GPPS,,NATAAU33XXXN201772001500514980144
2017/07/20 01:43:53.70,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE10044DF4,414D512044455647505030332020202059635ADE2017BC02,p002,,,H07KB43306950R25,NATAAU33XXXN201772001500514980144,,NATAAU33XXXN201772001500514980144
2017/07/20 01:44:03.83,GFRVTFRI.QUEUE,414D512044455647505030332020202059635ADE2017BC08,414D512044455647505030332020202059635ADE2017BC08,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 01:44:04. 7,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DF5,414D512044455647505030332020202059635ADE10044DF5,pacs.008.001.02,,H07KB43306950R25,H07KB44040F50S2JGPPS,,NATAAU33XXXN201772001500514980144,
2017/07/20 01:44:19.21,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017BC0B,414D512044455647505030332020202059635ADE2017BC0B,pacs.002.001.03,,H07KB43306950R25,1230123789422017720161500515056712,H07KB44040F50S2JGPPS,,NATAAU33XXXN201772001500514980144
2017/07/20 01:44:19.42,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044DF6,414D512044455647505030332020202059635ADE2017BC02,p002,,,H07KB43306950R25,NATAAU33XXXN201772001500514980144,,NATAAU33XXXN201772001500514980144
2017/07/20 01:44:32.53,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017BC0E,414D512044455647505030332020202059635ADE2017BC0E,p001,,,NATAAU33XXXN2017720301500515070204,,NATAAU33XXXN2017720301500515070204,
2017/07/20 01:44:33.89,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DF7,414D512044455647505030332020202059635ADE10044DF7,pacs.008.001.02,,H07KB4432EZ50R2P,H07KB4433NT50R01GPPS,,NATAAU33XXXN2017720301500515070204,
2017/07/20 01:44:52.69,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017BC11,414D512044455647505030332020202059635ADE2017BC11,pacs.002.001.03,,H07KB4432EZ50R2P,1230123789422017720501500515090413,H07KB4433NT50R01GPPS,,NATAAU33XXXN2017720301500515070204
2017/07/20 01:44:54.76,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE10044DF9,414D512044455647505030332020202059635ADE2017BC0E,p002,,,H07KB4432EZ50R2P,NATAAU33XXXN2017720301500515070204,,NATAAU33XXXN2017720301500515070204
2017/07/20 01:44:54.70,PFPCSEI.QUEUE,414D512044455647505030332020202059635ADE10044DF8,414D512044455647505030332020202059635ADE10044DF8,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 01:44:56.75,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002642,414D512044455647505030332020202059635ADE10044DF8,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T01:44:55.202Z_00027,,,,,
2017/07/20 01:44:56.80,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002649,414D5120444556504147303320202020596ECF3120002649,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T01:44:55.202Z_00027,,,,,
2017/07/20 01:44:57.17,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DFA,414D512044455647505030332020202059635ADE10044DFA,pacs.008.001.02,,H07KB4432EZ50R2P,H07KB44573U50V0DGPPS,,NATAAU33XXXN2017720301500515070204,
2017/07/20 01:44:57.84,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044DFB,414D512044455647505030332020202059635ADE2017BC0E,p002,,,H07KB4432EZ50R2P,NATAAU33XXXN2017720301500515070204,,NATAAU33XXXN2017720301500515070204
2017/07/20 01:45:13.14,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017BF02,414D512044455647505030332020202059635ADE2017BF02,pacs.002.001.03,,H07KB4432EZ50R2P,1230123789422017720101500515110717,H07KB44573U50V0DGPPS,,NATAAU33XXXN2017720301500515070204
2017/07/20 01:45:13.26,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044DFC,414D512044455647505030332020202059635ADE10044DFC,p002,,,H07KB4432EZ50R2P,NATAAU33XXXN2017720301500515070204,,NATAAU33XXXN2017720301500515070204
2017/07/20 01:45:26.43,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017BF05,414D512044455647505030332020202059635ADE2017BF05,p001,,,NATAAU33XXXN2017720241500515124090,,NATAAU33XXXN2017720241500515124090,
2017/07/20 01:45:26.76,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044DFD,414D512044455647505030332020202059635ADE10044DFD,pacs.008.001.02,,H07KB4526C350R0M,H07KB4526KF50R0QGPPS,,NATAAU33XXXN2017720241500515124090,
2017/07/20 01:45:47.10,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017CF02,414D512044455647505030332020202059635ADE2017CF02,pacs.002.001.03,,H07KB4526C350R0M,1230123789422017720441500515144771,H07KB4526KF50R0QGPPS,,NATAAU33XXXN2017720241500515124090
2017/07/20 01:45:48.23,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE10044E69,414D512044455647505030332020202059635ADE2017BF05,p002,,,H07KB4526C350R0M,NATAAU33XXXN2017720241500515124090,,NATAAU33XXXN2017720241500515124090
2017/07/20 01:45:48.06,PFPCSEI.QUEUE,414D512044455647505030332020202059635ADE10044E68,414D512044455647505030332020202059635ADE10044E68,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 01:45:48.98,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002742,414D512044455647505030332020202059635ADE10044E68,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T01:45:48.295Z_00028,,,,,
2017/07/20 01:45:49.01,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002749,414D5120444556504147303320202020596ECF3120002749,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T01:45:48.295Z_00028,,,,,
2017/07/20 01:45:50.29,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044E6A,414D512044455647505030332020202059635ADE10044E6A,pacs.008.001.02,,H07KB4526C350R0M,H07KB45503P50V1LGPPS,,NATAAU33XXXN2017720241500515124090,
2017/07/20 01:45:52.78,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044E6B,414D512044455647505030332020202059635ADE2017BF05,p002,,,H07KB4526C350R0M,NATAAU33XXXN2017720241500515124090,,NATAAU33XXXN2017720241500515124090
2017/07/20 01:46:05.61,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017CF05,414D512044455647505030332020202059635ADE2017CF05,pacs.002.001.03,,H07KB4526C350R0M,123012378942201772031500515163148,H07KB45503P50V1LGPPS,,NATAAU33XXXN2017720241500515124090
2017/07/20 01:46:06. 7,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044E6C,414D512044455647505030332020202059635ADE10044E6C,p002,,,H07KB4526C350R0M,NATAAU33XXXN2017720241500515124090,,NATAAU33XXXN2017720241500515124090
2017/07/20 01:52:05.40,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF312000274F,414D5120444556504147303320202020596ECF312000274F,pacs.008.001.05,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:52:04.871Z_00033,,,,,
2017/07/20 01:52:07.97,PFPCSEI.QUEUE,414D512044455647505030332020202059635ADE10044ED6,414D512044455647505030332020202059635ADE10044ED6,UNKNOWN MSGDEFIDR,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:52:04.871Z_00033,,,,,
2017/07/20 01:52:09.04,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002759,414D5120444556504147303320202020596ECF3120002759,pacs.002.001.06,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:52:04.871Z_00033,,,,,
2017/07/20 01:52:09.23,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044ED8,414D512044455647505030332020202059635ADE10044ED8,pacs.008.001.02,,H07KB5205CH5112L,H07KB52094W50T02GPPS,,20161109110033562894499,
2017/07/20 01:54:14.43,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002553,414D5120444556504147303320202020596ECF3120002553,pacs.008.001.05,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:54:13.909Z_00036,,,,,
2017/07/20 01:54:15.52,PFPCSEI.QUEUE,414D512044455647505030332020202059635ADE10044ED9,414D512044455647505030332020202059635ADE10044ED9,UNKNOWN MSGDEFIDR,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:54:13.909Z_00036,,,,,
2017/07/20 01:54:16.29,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF312000255D,414D5120444556504147303320202020596ECF312000255D,pacs.002.001.06,ANZBAU3L_A_TST01_ClrSttlmV001_2017-07-20T01:54:13.909Z_00036,,,,,

looking forward -thx

Re: timestamp difference between two different events with matching criteria

niketn — Sat, 22 Jul 2017 15:27:32 GMT

@shaganga, through requirement A, B , C seems like you want to extract two events and correlate them together. This can be done in single shot using stats as preferred method (there are other correlation techniques like transaction as well but might not be suitable for your scenario).

Few clarifications though, Your example with field names seems to have two MsgId fields in the same event, is that a typo, also first MsgId is same as CoreId? Is CoreId field available/same in both Event A and Event B
MsgId=414D512044455647505030332020202059635ADE2017BA02
MsgId=NATAAU33XXXN2017720591500514979309

It would have been more useful if you had added one sample each (with field names) of both Event A and Event B which are supposed to be correlated. Since with the requirements A, B, C the data provided is not being correlated.

Re: timestamp difference between two different events with matching criteria

niketn — Sat, 22 Jul 2017 15:57:23 GMT

This seems to be duplicate of question: https://answers.splunk.com/answers/557502/calculating-timestamp-difference-between-two-event.html

Re: timestamp difference between two different events with matching criteria

DalJeanis — Sun, 23 Jul 2017 21:39:45 GMT

Yes, it does - and it doesn't seem that the poster has actually done anything with the advice given there.

Re: timestamp difference between two different events with matching criteria

shaganga — Tue, 29 Sep 2020 15:02:52 GMT

@DalJeanis - i tried your scenario and responded that i am not getting the correct result with your query. all records showing timestamps difference as ZERO.. which is not correct. unfortunately i am unable to upload complete or more log due to restrictions here in textfield to upload. And also not sure whether u got my response or not.. hence i have raised a request with all details more.

@niketn - its not a typo. those are two different fields.. when the first time flows into that queue MsgID and MsgcoreID are same. when flow enters into other queue which stated above(Step a).. then MsgId of First queue matches with MsgCoreID with second queue which stated above(Step b).
iam unable to upload complete log due to restrictions. however pls find subsequent logs for more..

2017/07/20 01:54:16.66,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044EDB,414D512044455647505030332020202059635ADE10044EDB,pacs.008.001.02,,H07KB5414CK50B07,H07KB5416GI5140GGPPS,,20161109110033562894499,
2017/07/20 01:54:45. 7,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044EDC,414D512044455647505030332020202059635ADE10044EDC,pacs.008.001.02,,H07KB5205CH5112L,H07KB54451E50D0LGPPS,,20161109110033562894499,
2017/07/20 01:54:52.22,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044EDD,414D512044455647505030332020202059635ADE10044EDD,pacs.008.001.02,,H07KB5414CK50B07,H07KB54525J50O0QGPPS,,20161109110033562894499,
2017/07/20 01:56:37.66,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017DF02,414D512044455647505030332020202059635ADE2017DF02,p001,,,NATAAU33XXXN2017720371500515797461,,NATAAU33XXXN2017720371500515797461,
2017/07/20 01:56:38. 6,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044F48,414D512044455647505030332020202059635ADE10044F48,pacs.008.001.02,,H07KB5637IJ50Z1F,H07KB5637RQ50Z1JGPPS,,NATAAU33XXXN2017720371500515797461,
2017/07/20 01:56:53.91,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017DF05,414D512044455647505030332020202059635ADE2017DF05,pacs.002.001.03,,H07KB5637IJ50Z1F,1230123789422017720531500515813713,H07KB5637RQ50Z1JGPPS,,NATAAU33XXXN2017720371500515797461
2017/07/20 01:56:54.18,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE10044F4A,414D512044455647505030332020202059635ADE2017DF02,p002,,,H07KB5637IJ50Z1F,NATAAU33XXXN2017720371500515797461,,NATAAU33XXXN2017720371500515797461
2017/07/20 01:57:02.26,GFRVTFRI.QUEUE,414D512044455647505030332020202059635ADE2017DF08,414D512044455647505030332020202059635ADE2017DF08,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 01:57:02.38,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10044F4B,414D512044455647505030332020202059635ADE10044F4B,pacs.008.001.02,,H07KB5637IJ50Z1F,H07KB57029X5101TGPPS,,NATAAU33XXXN2017720371500515797461,
2017/07/20 01:57:12.91,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017DF0B,414D512044455647505030332020202059635ADE2017DF0B,pacs.002.001.03,,H07KB5637IJ50Z1F,1230123789422017720121500515832681,H07KB57029X5101TGPPS,,NATAAU33XXXN2017720371500515797461
2017/07/20 01:57:13. 0,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10044F4C,414D512044455647505030332020202059635ADE2017DF02,p002,,,H07KB5637IJ50Z1F,NATAAU33XXXN2017720371500515797461,,NATAAU33XXXN2017720371500515797461
2017/07/20 02:39:28.27,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017F302,414D512044455647505030332020202059635ADE2017F302,p001,,,NATAAU33XXXN2017720271500518367509,,NATAAU33XXXN2017720271500518367509,
2017/07/20 02:39:28.73,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10045298,414D512044455647505030332020202059635ADE10045298,pacs.008.001.02,,H07KC39287R5190R,H07KC3928JQ5190VGPPS,,NATAAU33XXXN2017720271500518367509,
2017/07/20 02:39:41.05,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017F402,414D512044455647505030332020202059635ADE2017F402,pacs.002.001.03,,H07KC39287R5190R,1230123789422017720401500518380542,H07KC3928JQ5190VGPPS,,NATAAU33XXXN2017720271500518367509
2017/07/20 02:39:41.59,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE1004529A,414D512044455647505030332020202059635ADE2017F302,p002,,,H07KC39287R5190R,NATAAU33XXXN2017720271500518367509,,NATAAU33XXXN2017720271500518367509
2017/07/20 02:39:52.42,GFRVTFRI.QUEUE,414D512044455647505030332020202059635ADE2017F502,414D512044455647505030332020202059635ADE2017F502,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 02:39:52.56,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE1004529B,414D512044455647505030332020202059635ADE1004529B,pacs.008.001.02,,H07KC39287R5190R,H07KC3952ES50115GPPS,,NATAAU33XXXN2017720271500518367509,
2017/07/20 02:40:01.43,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017F305,414D512044455647505030332020202059635ADE2017F305,pacs.002.001.03,,H07KC39287R5190R,123012378942201772001500518400999,H07KC3952ES50115GPPS,,NATAAU33XXXN2017720271500518367509
2017/07/20 02:40:01.51,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE1004529C,414D512044455647505030332020202059635ADE2017F302,p002,,,H07KC39287R5190R,NATAAU33XXXN2017720271500518367509,,NATAAU33XXXN2017720271500518367509
2017/07/20 02:40:08.63,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017F405,414D512044455647505030332020202059635ADE2017F405,p001,,,NATAAU33XXXN201772081500518408260,,NATAAU33XXXN201772081500518408260,
2017/07/20 02:40:08.75,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE1004529D,414D512044455647505030332020202059635ADE1004529D,pacs.008.001.02,,H07KC4008HO5191B,H07KC4008K65191FGPPS,,NATAAU33XXXN201772081500518408260,
2017/07/20 02:40:20.69,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017F505,414D512044455647505030332020202059635ADE2017F505,pacs.002.001.03,,H07KC4008HO5191B,1230123789422017720201500518420326,H07KC4008K65191FGPPS,,NATAAU33XXXN201772081500518408260
2017/07/20 02:40:20.87,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE1004529F,414D512044455647505030332020202059635ADE2017F405,p002,,,H07KC4008HO5191B,NATAAU33XXXN201772081500518408260,,NATAAU33XXXN201772081500518408260
2017/07/20 02:40:27.11,GFRVTFRI.QUEUE,414D512044455647505030332020202059635ADE2017F308,414D512044455647505030332020202059635ADE2017F308,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 02:40:27.19,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE100452A0,414D512044455647505030332020202059635ADE100452A0,pacs.008.001.02,,H07KC4008HO5191B,H07KC40274D5011PGPPS,,NATAAU33XXXN201772081500518408260,
2017/07/20 02:40:36.08,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017F408,414D512044455647505030332020202059635ADE2017F408,pacs.002.001.03,,H07KC4008HO5191B,1230123789422017720351500518435654,H07KC40274D5011PGPPS,,NATAAU33XXXN201772081500518408260
2017/07/20 02:40:36.15,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE100452A1,414D512044455647505030332020202059635ADE2017F405,p002,,,H07KC4008HO5191B,NATAAU33XXXN201772081500518408260,,NATAAU33XXXN201772081500518408260
2017/07/20 02:40:44.36,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017FD02,414D512044455647505030332020202059635ADE2017FD02,p001,,,NATAAU33XXXN2017720421500518442993,,NATAAU33XXXN2017720421500518442993,
2017/07/20 02:40:46.64,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE1004530C,414D512044455647505030332020202059635ADE1004530C,pacs.008.001.02,,H07KC4044BT5191F,H07KC4046AW5191XGPPS,,NATAAU33XXXN2017720421500518442993,
2017/07/20 02:40:56.45,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017FF02,414D512044455647505030332020202059635ADE2017FF02,pacs.002.001.03,,H07KC4044BT5191F,1230123789422017720561500518456033,H07KC4046AW5191XGPPS,,NATAAU33XXXN2017720421500518442993
2017/07/20 02:40:57.54,GTAPIO.QUEUE,414D512044455647505030332020202059635ADE1004530E,414D512044455647505030332020202059635ADE2017FD02,p002,,,H07KC4044BT5191F,NATAAU33XXXN2017720421500518442993,,NATAAU33XXXN2017720421500518442993
2017/07/20 02:40:57.45,PFPCSEI.QUEUE,414D512044455647505030332020202059635ADE1004530D,414D512044455647505030332020202059635ADE1004530D,UNKNOWN MSGDEFIDR,,,,,,
2017/07/20 02:40:58.82,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF3120002846,414D512044455647505030332020202059635ADE1004530D,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T02:40:57.258Z_00029,,,,,
2017/07/20 02:40:58.88,PTPCSRO.QUEUE,414D5120444556504147303320202020596ECF312000284D,414D5120444556504147303320202020596ECF312000284D,pacs.002.001.06,NATAAU33_A_TST01_ClrSttlmV001_2017-07-20T02:40:57.258Z_00029,,,,,
2017/07/20 02:40:59.35,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE1004530F,414D512044455647505030332020202059635ADE2017FD02,p002,,,H07KC4044BT5191F,NATAAU33XXXN2017720421500518442993,,NATAAU33XXXN2017720421500518442993
2017/07/20 02:41:00. 1,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10045310,414D512044455647505030332020202059635ADE10045310,pacs.008.001.02,,H07KC4044BT5191F,H07KC4059QC50F01GPPS,,NATAAU33XXXN2017720421500518442993,
2017/07/20 02:41:00.95,GTPIRO.QUEUE,414D512044455647505030332020202059635ADE10045311,414D512044455647505030332020202059635ADE10045311,pacs.008.001.02,,H07KC4044BT5191F,H07KC4100PC50T07GPPS,,NATAAU33XXXN2017720421500518442993,
2017/07/20 02:41:10.58,GFPIRSI.QUEUE,414D512044455647505030332020202059635ADE2017FF05,414D512044455647505030332020202059635ADE2017FF05,pacs.002.001.03,,H07KC4044BT5191F,1230123789422017720101500518470157,H07KC4100PC50T07GPPS,,NATAAU33XXXN2017720421500518442993
2017/07/20 02:41:10.69,GTNPSO.QUEUE,414D512044455647505030332020202059635ADE10045312,414D512044455647505030332020202059635ADE10045312,p002,,,H07KC4044BT5191F,NATAAU33XXXN2017720421500518442993,,NATAAU33XXXN2017720421500518442993
2017/07/20 02:41:18.53,GFSPII.QUEUE,414D512044455647505030332020202059635ADE2017FF08,414D512044455647505030332020202059635ADE2017FF08,p001,,,NATAAU33XXXN2017720171500518477267,,NATAAU33XXXN2017720171500518477267,

hope this helps.. looking forward !!

Re: timestamp difference between two different events with matching criteria

niketn — Mon, 24 Jul 2017 09:45:56 GMT

@shaganga, I dont need more log. Out of the sample data that you have provided, I just need two events one from scenario A (queue=GFSPII.QUEUE and MsgDefId=p001 event) and another from scenario B (queue=GTAPIO.QUEUE and MsgDefId=p002) with InstructionId and OriginalInstructionId matching for both of them.

Re: timestamp difference between two different events with matching criteria

DalJeanis — Mon, 24 Jul 2017 15:09:56 GMT

@shaganga - Yes, the excess data is making the discussion more complicated than necessary. I've closed the other question to avoid making this any more confusing.

Really, we don't need to see the full _raw data if the fields to be matched have already been extracted, just a table of the fields and their values.

Also, please ensure that you use the exact fieldname, correctly capitalized, in all your questions and comments. In the details you posted above, it is CoreId. elsewhere in the posts, you have typed MsgcoreID and MsgCoreID.

Re: timestamp difference between two different events with matching criteria

DalJeanis — Mon, 24 Jul 2017 15:28:44 GMT

Try this -

your search that brings back all records in all queues with fields listed below
| fields InstructionId MsgDefId timeStamp queueName 
| sort 0 InstructionId timeStamp
| stats min(timeStamp) as minTime, max(timeStamp) as maxTime, 
    min(MsgDefId) as minMsgDefId, list(MsgDefId) as MsgDefId, list(queueName) as queueName,
    earliest(queueName) as firstqueueName, latest(queueName) as lastqueueName by InstructionId
| nomv MsgDefId | nomv queueName
| eval duration = strptime(maxTime."%Y/%m/%d %H:%M:%S.%2Q") - strptime(minTime."%Y/%m/%d %H:%M:%S.%2Q")

Adjust the field capitalizations as needed.

Assumptions -

1) InstructionId is unique across the time of the query and uniquely identifies the transaction that needs to be matched together.

2) There is a plain old space between 2017/07/20 and 01:43:06.78
3) There is no particular queue that is any more important than any other queue for purposes of the search

Other Notes -

You can test minMsgDefId to verify that you have the first MsgDefId in the time frame of the search.

queueName gives you a space-separated list of all the queues the transaction went through, in timeStamp order. MsgDefId gives you those in order of execution as well.