Multistep transaction using stats

[knarinen3]

Hi, I have following stats table
key EventCode timestamp
5q9ptD2QRZGkIrv1hPD3Mg customerCreditTransferInitiationCompleted 2019-12-03T13:15:04.283Z
customerCreditTransferSettled 2019-12-03T13:15:04.275Z
customerCreditTransferInitiationProcessed 2019-12-03T13:15:03.820Z
customerCreditTransferInitiationReceived 2019-12-03T13:15:03.764Z

I would like to measure duration of each step. any ideas how to do it?

[adonio]

hello there,

there are many ways to achieve this, and to understand the best one, more information regarding your data is required. in the answer i also assumed you have teh key in each line / event.
below is one option that might meets your need, run it anywhere.

| makeresults count=1 
| eval data = "5q9ptD2QRZGkIrv1hPD3Mg customerCreditTransferInitiationCompleted 2019-12-03T13:15:04.283Z;;;5q9ptD2QRZGkIrv1hPD3Mg customerCreditTransferSettled 2019-12-03T13:15:04.275Z;;;5q9ptD2QRZGkIrv1hPD3Mg customerCreditTransferInitiationProcessed 2019-12-03T13:15:03.820Z;;;5q9ptD2QRZGkIrv1hPD3Mg customerCreditTransferInitiationReceived 2019-12-03T13:15:03.764Z"
| makemv delim=";;;" data
| mvexpand data
| rex field=data "(?<key>[^\s]+)\s+(?<EventCode>[^\s]+)\s+(?<time>.*+)"
| table time key EventCode
| rename COMMENT as "the above generates data below is the solution" 
| eval time_epoch = strptime(time, "%Y-%m-%dT%H:%M:%S.%3N")
| sort time_epoch
| streamstats range(time_epoch) as trans_duration by key

hope it helps

[knarinen3]

hi, the data provided was in stats format.
the raw data is like this:
{"container_name":"/beconsumer_KafkaConsumer.1.irz785be5dhsco32lqqtu51bh","source":"stdout","log":"2019-12-03 18:05:00 INFO EventLogger:%_ - Topic: cctiBusinessEvents-Nft01X, Key: 5q9ptD2QRZGkIrv1hPD3Mg, transactionTraceIdentification=ade1c48f-b51f-4b5c-8f17-ae1adcba15f4, paymentProduct=DEPFUND, entityId=f8b15f9d-44e8-48d2-b74b-a6c10c14682a, amount=1, eventCode=customerCreditTransferInitiationCompleted, channel=retail, Offset=226816, currency=DKK, businessEventId=POM-CustomerCreditTransferInitiationV07-2-1782729, instructionReceiptIdentification=PMTDKRG1239297, Partition=1, dateTime=2019-12-03T13:15:04.283Z","container_id":"0172589e4fab1e910305476a8090b66a0d0e5ce6fd8e076e99d51f333d9c45c4"}

{"source":"stdout","log":"2019-12-03 18:05:00 INFO EventLogger:%_ - Topic: cctiBusinessEvents-Nft01X, Key: 5q9ptD2QRZGkIrv1hPD3Mg, entityId=f8b15f9d-44e8-48d2-b74b-a6c10c14682a, eventCode=customerCreditTransferSettled, channel=retail, Offset=226815, businessEventId=POM-CustomerCreditTransferInitiationV07-2-1782726, instructionReceiptIdentification=65052\"}, Partition=1, dateTime=2019-12-03T13:15:04.275Z","container_id":"0172589e4fab1e910305476a8090b66a0d0e5ce6fd8e076e99d51f333d9c45c4","container_name":"/beconsumer_KafkaConsumer.1.irz785be5dhsco32lqqtu51bh"}

{"container_id":"0172589e4fab1e910305476a8090b66a0d0e5ce6fd8e076e99d51f333d9c45c4","container_name":"/beconsumer_KafkaConsumer.1.irz785be5dhsco32lqqtu51bh","source":"stdout","log":"2019-12-03 18:05:00 INFO EventLogger:%_ - Topic: cctiBusinessEvents-Nft01X, Key: 5q9ptD2QRZGkIrv1hPD3Mg, entityId=f8b15f9d-44e8-48d2-b74b-a6c10c14682a, eventCode=customerCreditTransferInitiationProcessed, channel=retail, Offset=226813, businessEventId=POM-CustomerCreditTransferInitiationV07-2-1782725, instructionReceiptIdentification=65052\"}, Partition=1, dateTime=2019-12-03T13:15:03.820Z"}

{"container_name":"/beconsumer_KafkaConsumer.1.irz785be5dhsco32lqqtu51bh","source":"stdout","log":"2019-12-03 18:05:00 INFO EventLogger:%_ - Topic: cctiBusinessEvents-Nft01X, Key: 5q9ptD2QRZGkIrv1hPD3Mg, transactionTraceIdentification=ade1c48f-b51f-4b5c-8f17-ae1adcba15f4, paymentProduct=DEPFUND, entityId=f8b15f9d-44e8-48d2-b74b-a6c10c14682a, amount=1, eventCode=customerCreditTransferInitiationReceived, channel=retail, Offset=226812, currency=DKK, businessEventId=POM-CustomerCreditTransferInitiationV07-2-1782709, instructionReceiptIdentification=PMTDKRG1239297, Partition=1, dateTime=2019-12-03T13:15:03.764Z","container_id":"0172589e4fab1e910305476a8090b66a0d0e5ce6fd8e076e99d51f333d9c45c4"}

and query I use to format it:
search query
| rex "Topic:\s+(?.?),\s+Key:\s+(?.?),"
| rex "entityId=(?\S+?),"
| rex "eventCode=(?\S+?),"
| rex "Partition=(?\S+?)"
| rex "dateTime=(?\S+?)\""
|stats values(topic) as Topic values(entityId) as EntityId list(eventCode) as EventCode list(dateTime) as timestamp by key