Splunk Search

How to create a new field based on sequence of events?

jlacal
Explorer

Howdy:
I'm a new Splunker so this may be a dumb question. I have looked around splunk>Answers and couldn't find a solution to my problem, So here it goes. Using Splunk Enterprise 6.1.2 on Mac OS X.

This is a follow-up to my earlier question:
http://answers.splunk.com/answers/148012/transaction-not-providing-all-events-in-target-range

My log files look like this:
...
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration
..
.... events during “aaCalibration” phase, all with datestamp
...
aaInfo
...
... events during “aaInfo” phase
...
aaMarks
...
... events during “aaMarks” phase, all with datestamp
...

I want to create a new field (let's call it "phase_name") that describes which "phase" of the program each event belongs to.

For example, in the case of:
aaRegistration
...
... events during “aaRegistration” phase, all with datestamp
...
aaCalibration

I want the new field where all events after " aaRegistration" and before " aaCalibration" to have "phase_name" = "aaRegistration"

My ultimate goal is to (hopefully) be able to retrieve all events belonging to the "aaRegistration" phase by using the "phase_name" field in a transaction.

Thank you.

= = = = = = =

Hello, Somesh:
Here's a set of actual events from my log files. Thank you for your assistance.

2013-12-30 16:11:16 966 LOG-1 aaRegistration
2013-12-30 16:11:17 006 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 007 LOG-1 vmstat:...
2013-12-30 16:11:17 043 LOG-1 free:...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 043 LOG-1 free: ...
2013-12-30 16:11:17 066 LOG-1 GetPre..
2013-12-30 16:11:17 470 LOG-1 Rob...
2013-12-30 16:11:17 490 LOG-1 _Send...
2013-12-30 16:11:17 603 LOG-1 Ro...
2013-12-30 16:11:17 790 LOG-1 _Send...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 Ro...
2013-12-30 16:11:17 800 LOG-1 _Send...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:11:17 810 LOG-1 Ro...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 649 LOG-1 CAUGHT AN ERROR: ...
2013-12-30 16:21:16 901 LOG-1 ----------------------------------------
2013-12-30 16:21:16 904 LOG-1 aaCalibration

Unfortunately I can not use [number of events] / [time lapse] between phases as a marker. I have to dynamically create a "counter" that says "aaRegistration starts here, all future events are assigned this label" until the parser encounters an event starting with "aa" that marks the beginning of the next phase.

Thanks.

1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try something like this:

your base search | rex field=_raw "\saa(?<Phase>\w+)$" | eval counter=if(isnotnull(Phase),1,0) | accum counter | eventstats values(Phase) as Phase by counter | fields - counter

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Try something like this:

your base search | rex field=_raw "\saa(?<Phase>\w+)$" | eval counter=if(isnotnull(Phase),1,0) | accum counter | eventstats values(Phase) as Phase by counter | fields - counter

ppablo
Retired

Hi @jlacal

You can accept @somesoni2's answer by clicking on the big check mark next to his response. You can also award more points (up vote) by clicking on the up facing arrow above the check mark. Glad you found a solution through Answers 🙂

Patrick

0 Karma

jlacal
Explorer

You are the man, it works. Thanks.
Now I need to find out how to give you points for providing the answer.

somesoni2
SplunkTrust
SplunkTrust

The search is just adding new fields (search time) and not doing any aggragation. If you just add "| table _raw Phase" at the end of the search, you can see each event have the a field called Phase which corresponds to aa<>.

jlacal
Explorer

Hello, Somesh:
Thank you for putting this together.
Unfortunately the code does not seem to do anything for me: I see the exact same timeline graph with my plain search code than by adding your code.
I'll play with your code further to see if I can tweak it to my needs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Could you share some actual events (mask all sensitive data)? Knowing the actual format of the data helps us provide accurate suggestions.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...