How to create a new field based on sequence of events?

[jlacal]

Howdy:
I'm a new Splunker so this may be a dumb question. I have looked around splunk>Answers and couldn't find a solution to my problem, So here it goes. Using Splunk Enterprise 6.1.2 on Mac OS X.

This is a follow-up to my earlier question:
http://answers.splunk.com/answers/148012/transaction-not-providing-all-events-in-target-range

My log files look like this:
...
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration
..
.... events during “aaCalibration” phase, all with datestamp
...
aaInfo
...
... events during “aaInfo” phase
...
aaMarks
...
... events during “aaMarks” phase, all with datestamp
...

I want to create a new field (let's call it "phase_name") that describes which "phase" of the program each event belongs to.

For example, in the case of:
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration

I want the new field where all events after " aaRegistration" and before " aaCalibration" to have "phase_name" = "aaRegistration"

My ultimate goal is to (hopefully) be able to retrieve all events belonging to the "aaRegistration" phase by using the "phase_name" field in a transaction.

Thank you.

= = = = = = =

Hello, Somesh:
Here's a set of actual events from my log files. Thank you for your assistance.

2013-12-30 16:11:16 966 LOG-1 aaRegistration
2013-12-30 16:11:17 006 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 043 LOG-1 free:...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 066 LOG-1 GetPre..
2013-12-30 16:11:17 470 LOG-1 Rob...
2013-12-30 16:11:17 490 LOG-1 _Send...
2013-12-30 16:11:17 603 LOG-1 Ro...
2013-12-30 16:11:17 790 LOG-1 _Send...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 _Send...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 901 LOG-1 ----------------------------------------
2013-12-30 16:21:16 904 LOG-1 aaCalibration

Unfortunately I can not use [number of events] / [time lapse] between phases as a marker. I have to dynamically create a "counter" that says "aaRegistration starts here, all future events are assigned this label" until the parser encounters an event starting with "aa" that marks the beginning of the next phase.

Thanks.

[somesoni2 · Accepted solution]

Try something like this:

your base search | rex field=_raw "\saa(?<Phase>\w+)$" | eval counter=if(isnotnull(Phase),1,0) | accum counter | eventstats values(Phase) as Phase by counter | fields - counter

[ppablo]

Hi @jlacal

You can accept @somesoni2's answer by clicking on the big check mark next to his response. You can also award more points (up vote) by clicking on the up facing arrow above the check mark. Glad you found a solution through Answers 🙂

Patrick

[jlacal]

You are the man, it works. Thanks.
Now I need to find out how to give you points for providing the answer.

[somesoni2]

The search is just adding new fields (search time) and not doing any aggragation. If you just add "| table _raw Phase" at the end of the search, you can see each event have the a field called Phase which corresponds to aa<>.

[jlacal]

Hello, Somesh:
Thank you for putting this together.
Unfortunately the code does not seem to do anything for me: I see the exact same timeline graph with my plain search code than by adding your code.
I'll play with your code further to see if I can tweak it to my needs.

[somesoni2]

Could you share some actual events (mask all sensitive data)? Knowing the actual format of the data helps us provide accurate suggestions.