Splunk Search

How to create a new field based on sequence of events?

jlacal
Explorer

Howdy:
I'm a new Splunker so this may be a dumb question. I have looked around splunk>Answers and couldn't find a solution to my problem, So here it goes. Using Splunk Enterprise 6.1.2 on Mac OS X.

This is a follow-up to my earlier question:
http://answers.splunk.com/answers/148012/transaction-not-providing-all-events-in-target-range

My log files look like this:
...
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration
..
.... events during “aaCalibration” phase, all with datestamp
...
aaInfo
...
... events during “aaInfo” phase
...
aaMarks
...
... events during “aaMarks” phase, all with datestamp
...

I want to create a new field (let's call it "phase_name") that describes which "phase" of the program each event belongs to.

For example, in the case of:
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration

I want the new field where all events after " aaRegistration" and before " aaCalibration" to have "phase_name" = "aaRegistration"

My ultimate goal is to (hopefully) be able to retrieve all events belonging to the "aaRegistration" phase by using the "phase_name" field in a transaction.

Thank you.

= = = = = = =

Hello, Somesh:
Here's a set of actual events from my log files. Thank you for your assistance.

2013-12-30 16:11:16 966 LOG-1 aaRegistration
2013-12-30 16:11:17 006 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 043 LOG-1 free:...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 066 LOG-1 GetPre..
2013-12-30 16:11:17 470 LOG-1 Rob...
2013-12-30 16:11:17 490 LOG-1 _Send...
2013-12-30 16:11:17 603 LOG-1 Ro...
2013-12-30 16:11:17 790 LOG-1 _Send...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 _Send...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 901 LOG-1 ----------------------------------------
2013-12-30 16:21:16 904 LOG-1 aaCalibration

Unfortunately I can not use [number of events] / [time lapse] between phases as a marker. I have to dynamically create a "counter" that says "aaRegistration starts here, all future events are assigned this label" until the parser encounters an event starting with "aa" that marks the beginning of the next phase.

Thanks.

1 Solution

somesoni2
Revered Legend

Try something like this:

your base search | rex field=_raw "\saa(?<Phase>\w+)$" | eval counter=if(isnotnull(Phase),1,0) | accum counter | eventstats values(Phase) as Phase by counter | fields - counter

View solution in original post

somesoni2
Revered Legend

Try something like this:

your base search | rex field=_raw "\saa(?<Phase>\w+)$" | eval counter=if(isnotnull(Phase),1,0) | accum counter | eventstats values(Phase) as Phase by counter | fields - counter

ppablo
Retired

Hi @jlacal

You can accept @somesoni2's answer by clicking on the big check mark next to his response. You can also award more points (up vote) by clicking on the up facing arrow above the check mark. Glad you found a solution through Answers 🙂

Patrick

0 Karma

jlacal
Explorer

You are the man, it works. Thanks.
Now I need to find out how to give you points for providing the answer.

somesoni2
Revered Legend

The search is just adding new fields (search time) and not doing any aggragation. If you just add "| table _raw Phase" at the end of the search, you can see each event have the a field called Phase which corresponds to aa<>.

jlacal
Explorer

Hello, Somesh:
Thank you for putting this together.
Unfortunately the code does not seem to do anything for me: I see the exact same timeline graph with my plain search code than by adding your code.
I'll play with your code further to see if I can tweak it to my needs.

0 Karma

somesoni2
Revered Legend

Could you share some actual events (mask all sensitive data)? Knowing the actual format of the data helps us provide accurate suggestions.

Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...