How to search between two lines?

[runiyal]

Hello All,

I have following lines in the log file -

Server8 runiyal 2023-01-12 09:48:41,880 INFO Plugin.DOCUMENT Bytes size from input stream : 2072823
server8 runiyal 2023-01-12 09:48:41,978 INFO Plugin.DOCUMENT File size after upload to temp folder: 2072823
server8 runiyal 2023-01-12 09:48:43,391 SUCCESS Plugin.DOCUMENT File size after notifying the docrepo : 2072823

I want to -
1. Search for the DocID in the end <2072823>; It should have SUCCESS written in line. (Line3)
2. It should then look at the above line with string "from input stream" for the same DocID (Line 1)
3. Reduce the timestamp from SUCCESS line (3) to the timestamp in line with the text "from input stream" (Line 1) - Result will be in seconds
4. Result should be in two columns: "DocID" and "Time Taken" (4)

Will appreciate your inputs on how this can be achieved. Thanks!

[gcusello]

Hi @runiyal,

I suppose that you already extracted the DocID field, otherwise you have to add a rex command before the stats command.

you could use the transaction command

index=your_index 1. (SUCCESS OR "from input stream" OR "from input stream")
| transaction DocID
| rename duration AS "Time Taken"
| table DocID "Time Taken"

that's very slow, so try this different approach:

index=your_index 1. (SUCCESS OR "from input stream" OR "from input stream")
| stats 
   earliest(_time) AS earliest
   latest(_time) AS latest
   BY DocID
| eval Time_Take=latest-earliest
| table DocID Time_Taken
| rename Time_Taken AS "Time Taken"

Ciao.

Giuseppe

[runiyal]

Thanks Giuseppe.. I have to extract the docID field too.

[gcusello]

Hi @runiyal,

in this case. you have to add a regex extraction to your search:

with transaction:

index=your_index 1. (SUCCESS OR "from input stream" OR "from input stream")
| rex "(?<DocID>\d+)$"
| transaction DocID
| rename duration AS "Time Taken"
| table DocID "Time Taken"

or with stats (better):

index=your_index 1. (SUCCESS OR "from input stream" OR "from input stream")
| rex "(?<DocID>\d+)$"
| stats 
   earliest(_time) AS earliest
   latest(_time) AS latest
   BY DocID
| eval Time_Take=latest-earliest
| table DocID Time_Taken
| rename Time_Taken AS "Time Taken"

You can test the regex at https://regex101.com/r/TgQtHA/1

Ciao.

Giuseppe