topic Re: How to extract two different fields on same timestamp based on the type of log statement? in Getting Data In

How to extract two different fields on same timestamp based on the type of log statement?

mlprasad79 — Tue, 31 Oct 2017 11:08:24 GMT

Hi there,
I have the following two different sample logger statements, the first statement written at the beginning of the process flow and the second logger is written at the end of the process flow.
1. [Info] 2017/09/09 12:00:00, 456 "Starting of the process"
2. [Info] 2017/09/09 12:00:00, 789 "End of the process".

Now, I want to extract 2 different fields on the time stamp, if the logger statement is "Starting of the process" I need to create beginTime field, and if the logger statement is "end of the process", I need to create endTime field, Please help.

Re: How to extract two different fields on same timestamp based on the type of log statement?

alemarzu — Tue, 31 Oct 2017 13:24:52 GMT

Hello there @mlprasad79

This might work.

... | rex "\]\s(?<beginTime>[\d\/\s:]+)(?=,\s\d+\s\"Starting)" | rex "\]\s(?<endTime>[\d\/\s:]+)(?=,\s\d+\s\"End)"

Hope it helps.

Re: How to extract two different fields on same timestamp based on the type of log statement?

mlprasad79 — Thu, 02 Nov 2017 12:02:21 GMT

Hi @Alemarzu,

Thanks for your reply,
The rex is giving result till this portion | rex "]\s(?[\d\/\s:]+)(?=,\s\d+\s , but if I append \"Starting, it is producing empty results, what went wong??

Re: How to extract two different fields on same timestamp based on the type of log statement?

alemarzu — Thu, 02 Nov 2017 13:08:43 GMT

My bad, square brackets at the beginning were not scaped. Its fixed now.

Re: How to extract two different fields on same timestamp based on the type of log statement?

mlprasad79 — Thu, 02 Nov 2017 15:06:37 GMT

Hi Alemarzu,

Here is my actual sample first and last logger statements,
[INFO ] 2017-11-02 10:58:16,071 - com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule 87675606-ddcc-4841-a925-96aac6a1a395-L7 MainflowOUT Exit the Ruleflow

[INFO ] 2017-11-02 10:58:16,071 - com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule 87675606-ddcc-4841-a925-96aac6a1a395-L7 MainflowIN Into the Ruleflow

where the highlighted is the string decided whther the logger is first statement or last.

Here is my search query,
((com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule * "MainflowIN Into the Ruleflow") OR (com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule * "MainflowOUT Exit the Ruleflow")) |rex "]\s(?P\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)(?=\s+-\s+\w+.\w+.\w+.\w+.\w+.\w+\s+\w+\s+ )"
if I try to hard code "MainflowIN" or "MainflowOUT" at the end of the query the results are blank, if I don't add this string results are coming but my ultimate goal is not achevied, please help.

Re: How to extract two different fields on same timestamp based on the type of log statement?

valiquet — Thu, 02 Nov 2017 16:28:21 GMT

It would be more efficient to do both extractions in a single regex.

Re: How to extract two different fields on same timestamp based on the type of log statement?

sbbadri — Tue, 29 Sep 2020 16:33:04 GMT

@mlprasad79

try this,

your search | rex field=_raw "[\S+\s+]\s(?P<end_time>\d+-\d+-\d+\s\d+:\d+:\d+\,\d+)\s+-\s+.+(?P<MainFlowOUT>MainflowOUT)" | rex field=_raw "[\S+\s+]\s(?P<begin_time>\d+-\d+-\d+\s\d+:\d+:\d+\,\d+)\s+-\s+.+(?P<MainFlowIN>MainflowIN)"

Re: How to extract two different fields on same timestamp based on the type of log statement?

mlprasad79 — Tue, 29 Sep 2020 16:37:20 GMT

Hi Badri,
Thanks for your reply,

The query is kinda working fine, now I am trying to figure out the response time which is end_time - begin_time.
For that I am using the below query,
--above query--|eval response_time=strptime(end_time,"%Y-%m-%d %H:%M:%S.%3N") - strptime(begin_time,"%Y-%m-%d %H:%M:%S.%3N") |table begin_time, end_time , response_time.

but the response_time column is coming empty, not sure what went wrong.

Re: How to extract two different fields on same timestamp based on the type of log statement?

sbbadri — Tue, 29 Sep 2020 16:33:46 GMT

Try this
strptime(end_time,”%Y/%m/%d %H:%M:%S, %3N”) and do the same thing for begin_time

Re: How to extract two different fields on same timestamp based on the type of log statement?

alemarzu — Fri, 03 Nov 2017 13:51:25 GMT

Oh I see what happened. The log sample that you provided at the beginnig is not the same like the one above.

Re: How to extract two different fields on same timestamp based on the type of log statement?

horsefez — Fri, 03 Nov 2017 14:01:15 GMT

Hi mlprasad,

look at this solution.

| rex field=_raw "\]\s+?((?<begin_time>.+?)\s\"Starting|(?<end_time>.+?)\s\"End)"

After that you are able to reformat the newly created fields further.