topic Re: How to capture start and End time interval for a particular file in Dashboards & Visualizations

How to capture start and End time interval for a particular file

aditsss — Tue, 29 Aug 2023 20:23:00 GMT

Hi Team,

I have one file CARS.HIERCTR for which I want to capture START and END DURATION

I am using below query:

ndex="600000304_d_gridgain_idx*" sourcetype =600000304_gg_abs_ipc2 | rex "\[(?<thread>Thread[^\]]+)\]" | transaction thread startswith="Reading Control-File /absin/CARS.HIERCTR." endswith="Completed Settlement file processing, CARS.HIER." | table duration

But I am not getting any result

Can someone guide me

Starting Logger - 2023-08-29 00:26:20.256 [INFO ] [pool-3-thread-1] ReadControlFileImpl - Reading Control-File /absin/CARS.HIERCTR.D082823.T001819

Ending logger - 2023-08-29 02:18:33.064 [INFO ] [Thread-34] FileEventCreator - Completed Settlement file processing, CARS.HIER.D082823.T020913 records processed: 135959

PLEASE GUIDE.

Re: How to capture start and End time interval for a particular file

bowesmana — Tue, 29 Aug 2023 23:23:52 GMT

Your Thread value is not extracted in the first data line and even if it was, it has a different value to the ending line. The first has 'thread-1' and the second Thread-34 although the first is actually pool-3-thread-1

Can you not use the D082823 ID in the CARS file description, e.g.

| rex "CARS\.HIER(CTR)?\.(?<CARS_ID>\w+)" | transaction CARS_ID startswith="Reading Control-File /absin/CARS.HIERCTR." endswith="Completed Settlement file processing, CARS.HIER." | table duration

Re: How to capture start and End time interval for a particular file

bowesmana — Tue, 29 Aug 2023 23:28:30 GMT

transaction command is not a good command to use for long transactions if you have a reasonable volume of data, as it will silently run out of memory and your results will be incomplete/wrong.

It is often better to use stats, e.g.

| stats min(_time) as Start max(_time) as Finish by CARS_ID | eval duration=Finish-Start

or if you have lots of events for the same ID that come before and after you could do

| stats min(eval(if(match(_raw, "Reading Control-File"), _time, null))) as Start max(eval(if(match(_raw, "Completed Settlement file processing"), _time, null))) as Finish by CARS_ID

but it will depend on your events - but this will be reliable