topic Re: How to combine multiple rex expressions and rename the field for an eval expression? in Splunk Search

How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Thu, 16 Jul 2015 05:32:47 GMT

Hi, I wonder if someone could help me please.

I'm currently using the following to extract certain fields contained with the events raw data.

| rex "Address Line 1=(?<address1>[^,]*)"  | rex "Address Line 2=(?<address2>[^,]*)"  | rex "Address Line 3=(?<address3>[^,]*)"  | rex "Address Line 4=(?<address4>[^,]*)"  | rex "Postcode=(?<postcode>[^,]*)"  |

But to cut down on the number of searches, I'm trying to join the rex expressions together, so using the inbuilt field extractor I've come up with the following:

rex "^(?:[^=\n]*=){6}(?P<Address_Line_1>[^=]+)[^ \n]* (?P<Address_Line_2>[^=]+)=,\s+(?P<Address_Line_3>\w+\s+\w+\s+\d+)=,\s+(?P<Address_Line_4>[^=]+)=,\s+\w+\s+(?P<Postcode>[^=]+)" |

But I'm having a little difficulty in replicating this part of the original rex expressions

 (?<address1>

where I'm renaming the field with the aim of then using an eval expression to create a combined Address field.

Could someone perhaps have a look at this please and offer some guidance on how may go about achieving this.
Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

jeffland — Thu, 16 Jul 2015 06:49:25 GMT

A sample of such an event would be helpful. Also, is there a reason you want to use rex instead of automatic field extractions?

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Thu, 16 Jul 2015 07:11:44 GMT

Hi @jeffland, thank you for taking the time to reply to my post.

Unfortunately I'm unable to provide you with event details because of their confidential nature, but the address lines are in the following format:

Address Line 1=1The Street, Address Line 2=The Town, Address Line 3=, Address Line 4=The City, Postcode=AB12CD

I'm not sure whether this helps.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

dineshraj9 — Thu, 16 Jul 2015 08:24:12 GMT

Try this -

| rex "Address Line 1=(?[^,]*)\,\s*Address Line 2=(?[^,]*)\,\s*Address Line 3=(?[^,]*)\,\s*Address Line 4=(?[^,]*)\,\s*Postcode=(?[^,]*)"

Re: How to combine multiple rex expressions and rename the field for an eval expression?

jeffland — Thu, 16 Jul 2015 09:00:01 GMT

Your "sample" is just what I wanted, of course you don't have to post actual content - just the form of it. Have a look at this:

[^\=]*\=(?<Address_Line_1>[^\,]*)[^\=]*\=(?<Address_Line_2>[^,]*)[^\=]*\=(?<Address_Line_3>[^\,]*)[^\=]*\=(?P<Address_Line_4>[^\,]*)[^\=]*\=(?<Postcode>[\w]*)

I was curious to see a sample because the regex the field extractor built looked weird.

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Thu, 16 Jul 2015 11:04:50 GMT

Hi thank you for taking the time to reply to my post.

I've tried the expression you kindly provided, and unfortunately I'm receiving the following error:

Error in 'rex' command: Encountered the following error while compiling the regex 'Address Line 1=(?[^,]*)\,\s*Address Line 2=(?[^,]*)\,\s*Address Line 3=(?[^,]*)\,\s*Address Line 4=(?[^,]*)\,\s*Postcode=(?[^,]*)': Regex: unrecognized character after (? or (?-

In additon, and forgive me, but am I right in thinking that this won't for example find 'Address Line 1' and then change it to 'address1' as per my original query.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Thu, 16 Jul 2015 11:07:17 GMT

Hi @jeffland, thank you for coming back to me with this.

The expression works, but it doesn't change for example 'Address Line 1' to 'address1' as per my original post.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

dineshraj9 — Thu, 16 Jul 2015 11:58:38 GMT

| rex "Address Line 1=(?<addressline1>[^,]*)\,\s*Address Line 2=(?<addressline2>[^,]*)\,\s*Address Line 3=(?<addressline3>[^,]*)\,\s*Address Line 4=(?<addressline4>[^,]*)\,\s*Postcode=(?postcode[^,]*)"

Re: How to combine multiple rex expressions and rename the field for an eval expression?

dineshraj9 — Thu, 16 Jul 2015 12:17:57 GMT

| rex "Address Line 1=(?<addressline1>[^,]*)\,\s*Address Line 2=(?<addressline2>[^,]*)\,\s*Address Line 3=(?<addressline3>[^,]*)\,\s*Address Line 4=(?<addressline4>[^,]*)\,\s*Postcode=(?postcode[^,]*)"

Re: How to combine multiple rex expressions and rename the field for an eval expression?

jeffland — Thu, 16 Jul 2015 13:09:17 GMT

Ok, then just rename the capturing group in question - i.e. rename

[^\=]*\=(?<Address_Line_1>[^\,]*) ...

[^\=]*\=(?<address1>[^\,]*) ...

Re: How to combine multiple rex expressions and rename the field for an eval expression?

woodcock — Thu, 16 Jul 2015 13:14:07 GMT

Just swap out the names as you see fit like this:

Before:

[^\=]*\=(?<Address_Line_1>[^\,]*)[^\=]*\=(?<Address_Line_2>[^,]*)[^\=]*\=(?<Address_Line_3>[^\,]*)[^\=]*\=(?P<Address_Line_4>[^\,]*)[^\=]*\=(?<Postcode>[\w]*)

After:

[^\=]*\=(?<address1>[^\,]*)[^\=]*\=(?<address2>[^,]*)[^\=]*\=(?<address3>[^\,]*)[^\=]*\=(?<address4>[^\,]*)[^\=]*\=(?<Postcode>[\w]*)

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Tue, 29 Sep 2020 06:43:06 GMT

Hi @woodcock, thank you for coming back to me with this.

Unfortunately this doesn't work.

If you look at my original post, the expression was as follows: rex "Address Line 1=(?[^,]*)".

With this, the expression searches for 'Address_Line_1' which is how the field is formatted in the raw data and then the second part renames this to 'adddress1'.

In the query you kindly provided, this only looks for 'Address_Line_1' it doesn't rename this.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Tue, 29 Sep 2020 06:43:09 GMT

Hi @jeffland, thank you for coming back to me, but unfortuantely this doens't work, because the expression is not looking for 'Address_Line_1' before renaming it.

If you look at my original expression: rex "Address Line 1=(?[^,]*)" the first part looks for the 'Address_Line_1' field, then it renames this to 'address1' which I can then pick up lateron in my query to pull all the address details together.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

woodcock — Thu, 16 Jul 2015 13:35:58 GMT

Chris, did you actually try the After command? I believe you are misunderstanding the RegEx. The string between the angle-brackets is the NAME of the captured group. It seems that you are assuming that because the literal string "Address Line " is not present in my RegEx, that it cannot work, but this is incorrect. I can match it based on the placement and ordering of the equals-signs also, which is what the RegEx is doing. Just try it.

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Thu, 16 Jul 2015 13:50:37 GMT

Hi @woodcock, thnak you for this.

Yes I did try the expression, but as I say, isn't extracting the data.

Many thanks and kind regards

Chris

Re: How to combine multiple rex expressions and rename the field for an eval expression?

jeffland — Thu, 16 Jul 2015 14:00:58 GMT

That's not how rex works. The command extracts something from an existing string and can place selected parts of that under any name in a field.

I just tried to smarten up your initial regex by not using the rigid "Address Line 1=", but you could of course keep that rigid format:

Address\sLine\s1\=(?<address1>[^\,]*)Address\sLine\2\=(?<address2>[^,]*)...

Re: How to combine multiple rex expressions and rename the field for an eval expression?

woodcock — Thu, 16 Jul 2015 14:04:03 GMT

How can that be? I copied the "Before" RegEx directly from the answer by @jeffland under which you said this:

The expression works, but it doesn't change for example 'Address Line 1' to 'address1' as per my original post.

So I modified the "working" solution to do that last part. You have got to get your stories straight or provide sample data or nobody is going to be able to help you.

Re: How to combine multiple rex expressions and rename the field for an eval expression?

ppablo — Thu, 16 Jul 2015 18:48:13 GMT

Hi @dineshraj9

Please don't post multiple answers on one question. If you're trying to add more content, just comment below your original answer. Also, it's helpful if you explain your searches to the user (and everyone else reading this post) rather than just copying and pasting it without context. I've converted your other 2 answers under your first one already, so something to keep in mind for the future. Thanks.

Re: How to combine multiple rex expressions and rename the field for an eval expression?

dineshraj9 — Fri, 17 Jul 2015 03:43:02 GMT

@ppablo : I really don't understand why I keep getting messages that moderator would review the post and once they approve it will show up.
Ya will ensure that I explain my answers. But please improve the portal to not crash when uploading answers and moderators to be quick in reviewing the answers and uploading it..
Thanks!!!

Re: How to combine multiple rex expressions and rename the field for an eval expression?

IRHM73 — Fri, 17 Jul 2015 06:20:20 GMT

Hi @ woodcock, thank you for coming back to me with this.

Firstly my apologies if you feel my stories aren't straight, personally I don' t feel this the case, as this has changed since my initial post. I did also provide sample data to @jefferson who found it more than adequate, see below:

Address Line 1=1The Street, Address Line 2=The Town, Address Line 3=, Address Line 4=The City, Postcode=AB12CD

I am very new to Splunk so I appreciate that rex not be the correct command to work, but it seems to have worked fine so far.

I'll try to explain in simpler terms what I'm trying to acheive and hopefully this helps.

My original expression was | rex "Address Line 1=(?[^,]*)"........... The first part of the expression searches my 'Raw Data' for the field 'Address Line 1". It then assigns the variable 'address1' to this. This variable is used later on in my full search.

As mentioned in my original post, rather than having multiple searcheds i.e '| rex "Address....' I wanted to bring all the elements of the address into one rex expression.

Forgive me but from the testing I've done the expressions you kindly provided, certainly from my testing don't search for the 'Address Line 1' field to assign the variable 'address1' to it.

I hope this helps.

Many thanks and kind regards

Chris