topic Re: How to edit my props.conf and transforms.conf to extract field names and values from my events? in Splunk Search

How to edit my props.conf and transforms.conf to extract field names and values from my events?

edrivera3 — Fri, 16 Oct 2015 20:48:14 GMT

I want to extract the field names and field values of my events.
My event looks like this:

Step:  1000
Result: blabkbk
Actual: blabblabl
Step:  1100
Result: blabkbk
Actual: blabblabl

I want the field name to be "1000" and "1100"and the respective field values to be everything below them. This is how I set up my props.conf and transforms.conf, but I am not extracting anything. I appreciate your help.

props.conf

[<mysourcetype>]
REPORT-step_num = step_num

transforms.conf

[step_num]
REGEX = STEP:\s+(?<_KEY_1>/d+)\n(?<_VAL_1>[\w\W\n]+?)\nSTEP

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

Richfez — Fri, 16 Oct 2015 22:34:52 GMT

Just a note: STEP does not equal Step.

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

sundareshr — Tue, 29 Sep 2020 07:37:52 GMT

How have you configured you LINE-BREAKER? One suggestion is to set your LINE-BREAKER as follows

SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Step:

Then you can define you AUTO-EXTRACT to be something like this

REGEX = ([^\W]+):=\s+([^\W]+
FORMAT = $1::$2

Just a thought. Not sure I understand your questions. What do you expect your end result to look like? How have you set up your event line-breaking? From your example, would you like Step: 1000 Result: blabkbk Actual: blabbk to be one event and so on?

If that's how you would like to see the events, then try adding the following to your props.conf

SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = Step:

And for field extraction (AUTO-EXTRACT) you could use something like

REGEX = ([^\W]+):=\s+([^\W]+) 
FORMAT = $1::$2

The above is just a general idea to get you started. Hopefully this helps.

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

woodcock — Sat, 17 Oct 2015 22:12:04 GMT

Like this in transforms.conf:

REGEX=(?ms)^Step:\s*([^\r\n]+)[\r\n]+(.*?)(?=\Z|[\r\n]+Step:)
CLEAN_KEYS = false
FORMAT = $1::$2
MV_ADD = 1

You should be able to test it on directly in search like this:

... | rex max_match=0 "(?ms)^Step:\s*(?<_KEY_1>[^\r\n]+)[\r\n]+(?<_VAL_1>.*?)(?=\Z|[\r\n]+Step:)"

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

edrivera3 — Thu, 22 Oct 2015 14:37:57 GMT

I tried the regex and it doesn't extract anything. Also I read that the name of the group must start with an alpha value.

I tried the following first regex inline and it worked. I tried the second one and Splunk displayed an error related with the alpha value issue.

... | rex "STEP:\s+\d+\n(?<myvalue>[\w\W\n]+?)\nSTEP" 
... | rex "STEP:\s+\d+\n(?<1myvalue>[\w\W\n]+?)\nSTEP"

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

edrivera3 — Thu, 22 Oct 2015 14:42:12 GMT

The event cannot be divided because there are relevant information at the beginning of the event related to all the steps. If I divided the steps of the event, then the information in this step would not have any reference of what's about and they would be useless.

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

woodcock — Thu, 22 Oct 2015 17:21:34 GMT

The _KEY_1 and _VAL_1 are special (valid) exceptions:

http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/Configureindex-timefieldextraction#Add_a_regex_stanza_for_the_new_field_to_transforms.conf

I was guessing that it could be tested inline but perhaps it cannot. In any case, the transforms.conf will work. The (ms) part is critical.

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

edrivera3 — Thu, 22 Oct 2015 18:20:50 GMT

Yes, but I was talking about the invalid field names which are "1000" and "1100" in my example.

Re: How to edit my props.conf and transforms.conf to extract field names and values from my events?

woodcock — Thu, 22 Oct 2015 18:55:33 GMT

Those names are valid for Search Time, provided that you tell splunk not to clean them. To do that, be sure to set this:

CLEAN_KEYS = false

http://www.splunk.com/base/Documentation/6.3.0/Admin/Transformsconf

NOTE: This attribute is only valid for search-time field extractions.
Optional. Controls whether Splunk "cleans" the keys (field names) it extracts at search time. "Key cleaning" is the practice of replacing any non-alphanumeric characters (characters other than those falling between the a-z, A-Z, or 0-9 ranges) in field names with underscores, as well as the stripping of leading underscores and 0-9 characters from field names.
Add CLEAN_KEYS = false to your transform if you need to extract field names that include non-alphanumeric characters, or which begin with underscores or 0-9 characters.
Defaults to true.