Getting Data In

How to extract two different fields on same timestamp based on the type of log statement?

mlprasad79
New Member

Hi there,
I have the following two different sample logger statements, the first statement written at the beginning of the process flow and the second logger is written at the end of the process flow.
1. [Info] 2017/09/09 12:00:00, 456 "Starting of the process"
2. [Info] 2017/09/09 12:00:00, 789 "End of the process".

Now, I want to extract 2 different fields on the time stamp, if the logger statement is "Starting of the process" I need to create beginTime field, and if the logger statement is "end of the process", I need to create endTime field, Please help.

0 Karma

horsefez
Motivator

Hi mlprasad,

look at this solution.

| rex field=_raw "\]\s+?((?<begin_time>.+?)\s\"Starting|(?<end_time>.+?)\s\"End)"

After that you are able to reformat the newly created fields further.

0 Karma

sbbadri
Motivator

@mlprasad79

try this,

your search | rex field=_raw "[\S+\s+]\s(?P<end_time>\d+-\d+-\d+\s\d+:\d+:\d+\,\d+)\s+-\s+.+(?P<MainFlowOUT>MainflowOUT)" | rex field=_raw "[\S+\s+]\s(?P<begin_time>\d+-\d+-\d+\s\d+:\d+:\d+\,\d+)\s+-\s+.+(?P<MainFlowIN>MainflowIN)"

0 Karma

mlprasad79
New Member

Hi Badri,
Thanks for your reply,

The query is kinda working fine, now I am trying to figure out the response time which is end_time - begin_time.
For that I am using the below query,
--above query--|eval response_time=strptime(end_time,"%Y-%m-%d %H:%M:%S.%3N") - strptime(begin_time,"%Y-%m-%d %H:%M:%S.%3N") |table begin_time, end_time , response_time.

but the response_time column is coming empty, not sure what went wrong.

0 Karma

sbbadri
Motivator

Try this
strptime(end_time,”%Y/%m/%d %H:%M:%S, %3N”) and do the same thing for begin_time

0 Karma

alemarzu
Motivator

Hello there @mlprasad79

This might work.

... | rex "\]\s(?<beginTime>[\d\/\s:]+)(?=,\s\d+\s\"Starting)" | rex "\]\s(?<endTime>[\d\/\s:]+)(?=,\s\d+\s\"End)"

Hope it helps.

0 Karma

mlprasad79
New Member

Hi @Alemarzu,

Thanks for your reply,
The rex is giving result till this portion | rex "]\s(?[\d\/\s:]+)(?=,\s\d+\s , but if I append \"Starting, it is producing empty results, what went wong??

0 Karma

valiquet
Contributor

It would be more efficient to do both extractions in a single regex.

0 Karma

alemarzu
Motivator

My bad, square brackets at the beginning were not scaped. Its fixed now.

0 Karma

mlprasad79
New Member

Hi Alemarzu,

Here is my actual sample first and last logger statements,
[INFO ] 2017-11-02 10:58:16,071 - com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule 87675606-ddcc-4841-a925-96aac6a1a395-L7 MainflowOUT Exit the Ruleflow

[INFO ] 2017-11-02 10:58:16,071 - com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule 87675606-ddcc-4841-a925-96aac6a1a395-L7 MainflowIN Into the Ruleflow

where the highlighted is the string decided whther the logger is first statement or last.

Here is my search query,
((com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule * "MainflowIN Into the Ruleflow") OR (com.aetna.eie.vtwoprovider.helper.util VTwoProviderRule * "MainflowOUT Exit the Ruleflow")) |rex "]\s(?P\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)(?=\s+-\s+\w+.\w+.\w+.\w+.\w+.\w+\s+\w+\s+ )"
if I try to hard code "MainflowIN" or "MainflowOUT" at the end of the query the results are blank, if I don't add this string results are coming but my ultimate goal is not achevied, please help.

0 Karma

alemarzu
Motivator

Oh I see what happened. The log sample that you provided at the beginnig is not the same like the one above.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...