topic Re: How to extract key, field name, and value with regex? in Splunk Search

How to extract key, field name, and value with regex?

tcmarquesi — Tue, 29 Sep 2020 11:57:53 GMT

I'm wondering if somebody had faced this freaking behavior.

I wanna extract both key, the field name, and its value from my (pretty uncommon) log and, in order to this I did the following:

In first place I made the search bellow just to test the regex, and it's working perfectly.

... | rex max_match=0 field=_raw "(?<test1>\w+)$.+$=(?<test2>[^\(].*)[\n|\r]"

I then replaced the test1 and test2 tags by _KEY_1 and _VAL_1 to assign properly each matched group to key and value as I wanted.

... | rex max_match=0 field=_raw "(?<_KEY_1>\w+)$.+$=(?<_VAL_1>[^\(].*)[\n|\r]"

From here ahead the extraction didn't work anymore.

So, had someone handled successfully same problem using this _KEY_1 and _VAL_1 tags? It seems like a bug for me.

Thanks in advance,

Tiago

Re: How to extract key, field name, and value with regex?

rjthibod — Thu, 01 Dec 2016 20:53:19 GMT

Leading underscores on field names is a no-no. Splunk uses leading underscores on field names for special / hidden fields.

Try renaming your fields to something with no leading underscore.

Re: How to extract key, field name, and value with regex?

rjthibod — Thu, 01 Dec 2016 20:54:56 GMT

Here is a link with more details about internal fields. http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Usedefaultfields

Re: How to extract key, field name, and value with regex?

sundareshr — Thu, 01 Dec 2016 20:56:02 GMT

Splunk regex does not like _ in field names. Having said that, have your looked at the extract command, that may be a better options.

... | extract kvdelim="=" pairdelim="\n"

http://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/Extract

Re: How to extract key, field name, and value with regex?

snoobzilla — Thu, 01 Dec 2016 20:56:24 GMT

You need to do this using a field transform and reference that transform in a field extraction. I can get these working on regex101.com but have not had luck using them inline.

See https://answers.splunk.com/answers/126754/transforms-field-value-extract-not-fully-working.html

Re: How to extract key, field name, and value with regex?

tcmarquesi — Thu, 01 Dec 2016 20:56:57 GMT

Just few additional comments:

I need to use regex because my log is a little unusual, it can't be automatically parsed.

I don't want to change my log with sed or something like that, is important to me keep it original.

In fact I intend to implement it in transforms.conf. I made the question using the SPL search because it behaved equally and it's easier to be reproduced.

Regards,

Tiago

Re: How to extract key, field name, and value with regex?

tcmarquesi — Fri, 02 Dec 2016 09:34:01 GMT

Thanks, but you missed my log is not that simple. Between key and value there is some text like "(foo 12)=". So I have to use regex, extract is ineffective.

Re: How to extract key, field name, and value with regex?

snoobzilla — Fri, 02 Dec 2016 11:25:47 GMT

Yes, I have done this, not with a variable delimiter, but I think a field transform will work.

I used this for logs with ]:[ key-value delimiter and ] [ as pair delimiter, e.g. [KEY1]:[VALUE1] [KEY2]:[VALUE2] [KEY3.....

From webui for example above...

Create Transform...

Fields-->Field Transformations--New
Regular Expression: \[([a-zA-Z0-9_]*?)\]\:\[([^\]]*?)\]
Source Key: _raw
Format: $1::$2

Create Extract
Then create new field extract, choose Type of transform, and point to the transform you created.

Tip: use regex101.com or equivalent to test your regex... it will work there and in transform but I get errors using this inline.

Re: How to extract key, field name, and value with regex?

tcmarquesi — Tue, 29 Sep 2020 11:58:16 GMT

Just to stay everybody in the same page, using "_" is not a problem, indeed both _KEY_foo and _VAL_bar are reserved tags in order to allow splunk find the field name a its value into the text, as in docs.

http://docs.splunk.com/Documentation/Splunk/6.5.0/Data/Configureindex-timefieldextraction#Add_a_regex_stanza_for_the_new_field_to_transforms.conf

Re: How to extract key, field name, and value with regex?

tcmarquesi — Fri, 02 Dec 2016 12:38:20 GMT

I'd done this but through transforms.conf. Indeed I can see my stanza through UI.

About the regex, I tested it exhaustively in both regex101.com and regexr.com/v1, and it's working perfectly.

Re: How to extract key, field name, and value with regex?

snoobzilla — Fri, 02 Dec 2016 12:54:22 GMT

Did you try method above with your rex without named capturing groups at all?... e.g.

(\w+)\(.+\)=([^\(].*)[\n|\r]

Note the Format field in transform: $1::$2

Re: How to extract key, field name, and value with regex?

snoobzilla — Fri, 02 Dec 2016 12:55:44 GMT

Or maybe

(\w+?)\(.+\)=([^\(].*?)[\n|\r]

Re: How to extract key, field name, and value with regex?

tcmarquesi — Fri, 02 Dec 2016 12:58:47 GMT

Yes, I did. It was my starting point.

This issue really seems as a bug for me...

Re: How to extract key, field name, and value with regex?

snoobzilla — Fri, 02 Dec 2016 13:11:57 GMT

Bummer. You may be right, may be a limitation.

Assume you saw this... https://answers.splunk.com/answers/133561/multiple-key-value-pair-extraction.html

Good luck.

Re: How to extract key, field name, and value with regex?

tcmarquesi — Fri, 02 Dec 2016 15:21:00 GMT

Thanks all help. 🙂

Re: How to extract key, field name, and value with regex?

aguthrie1190 — Thu, 07 Feb 2019 22:03:00 GMT

Late to the party here, but I had a similar need to this and saw that this question hadn't been answered. Basically do your extractions, then use {} in an eval to have a variable fieldname.

| gentimes start=-2
| eval _raw="extract"+starttime+" this"+endtime
| rex field=_raw "(?<field_name>extract[0-9]+)\s(?<field_value>this[0-9]+)"
| eval {field_name}=field_value

Then if you care, you can get rid of the placeholder fields:

| gentimes start=-2
| fields - *human
| eval _raw="extract"+starttime+" this"+endtime
| rex field=_raw "(?<field_name>extract[0-9]+)\s(?<field_value>this[0-9]+)"
| eval {field_name}=field_value
| fields - field_name field_value

These searches should run anywhere. The idea came from here https://answers.splunk.com/answers/103700/how-do-i-create-a-field-whose-name-is-the-value-of-another-field-like-backticks-or-eval-in-other-languages.html.

Re: How to extract key, field name, and value with regex?

irenefdezbb — Fri, 02 Oct 2020 11:46:38 GMT

Maybe, not working with _KEY_1 and _VALUE_1 because of splunk reserves the fields beginning with _ for your own settings, if I remember correctly.