topic Re: How to extract two fields from a group in Splunk Search

How to extract two fields from a group?

kmhanson — Thu, 13 Apr 2023 15:52:33 GMT

I am new to Regex expressions and trying to figure them out.

I am trying to extract two sections of the following log field:

5002:fromhost=999.99.99.99:fromport=3299:sid=92ac3498-d95d-11ed-af19-92eb6037d638:respcode=OK:resptime=7:node=999999ss03:nodePort=5002:cosId=asasasa

I want the IP address that shows after fromhost and the COSID value asasasa at the end of the field and not having much luck

Re: How to extract two fields from a group

ITWhisperer — Thu, 13 Apr 2023 13:40:43 GMT

fromhost=(?<fromhost>[^:]+).*cosId=(?<cosid>.*)

https://regex101.com/r/rZq5Gn/1

Re: How to extract two fields from a group

kmhanson — Thu, 13 Apr 2023 14:30:55 GMT

If I just want the IP address and not the COSID, what do I cut out? Turns out COSID isn't always there

Re: How to extract two fields from a group

ITWhisperer — Thu, 13 Apr 2023 14:35:39 GMT

Use two separate expressions

fromhost=(?<fromhost>[^:]+)

cosId=(?<cosid>.*)

That way, you will get the field if the anchor matches, and it will be null if the anchor isn't found

Re: How to extract two fields from a group

dtsariapkin — Thu, 13 Apr 2023 14:38:50 GMT

That is very elegant solution by @ITWhisperer here. Depending on how many logs you have and how far you go with your REGEX learning you might want to start doing a bit more defined groups too e.g.:
fromhost=(?<fromhost>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*cosId=(?<cosid>[^\s]+)

Which now looks for exact pattern of the IP address. Or something more convoluted but doing same thing:
fromhost=(?<fromhost>(\d{1,3}(\.)?){4}).+cosId=(?<cosid>[^\s]+)

Where numbers placed in curly brackets tell you how much the preceding pattern would repeat:
e.g {1,3} from one to 3 times. As in \d{1,3} means any digit one or three times like in the IP address.

Why things like that important? Is because the more you start work with Splunk and the more events you parse that way. The more patterns of .* might start to impact you and hogging your CPU. Seen too many issues when the patterns start to go out of control.

Good luck and welcome to the wonderful world of Regular Expressions.

Re: How to extract two fields from a group

kmhanson — Thu, 13 Apr 2023 14:39:43 GMT

so is the full command: | rex field=port mode=sed fromhost=(?<fromhost>[^:]+)

Re: How to extract two fields from a group

ITWhisperer — Thu, 13 Apr 2023 14:44:02 GMT

No - mode=sed is for stream editing, which is not required when you are just extracting fields, and assuming you have already extract the port field holding all this information (which was clear from your original post)

| rex field=port "fromhost=(?<fromhost>[^:]+)"

Re: How to extract two fields from a group

kmhanson — Thu, 13 Apr 2023 14:44:57 GMT

I have hundreds of thousands of logs. I can put it in excel but would rather get it done in sprint so the rest of the team can run the same command. It is more important to get that ip address, the COSID isn't so important

Re: How to extract two fields from a group

dtsariapkin — Thu, 13 Apr 2023 15:06:56 GMT

@kmhanson
1) If you adamant in doing it all in single expression. You can do it like that:
fromhost=(?<fromhost>[^:]+)(.*cosId=(?<cosid>.*))?

Notice I put second part in brackets and put question mark at the end. That means that whatever is in parenthesis before can match once or not match at all.

2) stick with the basic mode first. SED is for replacing things.
3) And you do not want field port do you? Not sure it does not exactly state that. Or I am being stupid.
4) So I would assume you will be extracting from RAW log -> Original log.

And your final test search would be:

| rex field=_raw "fromhost=(?<fromhost>[^:]+)" | rex field=_raw "cosId=(?<cosid>.*)"

OR!

| rex field=_raw "fromhost=(?<fromhost>[^:]+)(.*cosId=(?<cosid>.*))?"

All in all in this command you say from which field you want to extract. "_raw" gives you the whole event. And then you place Regular expression inside the quotes.

If you find any of the solutions good. Do not forget to mark it as answered/solved.

Re: How to extract two fields from a group

kmhanson — Thu, 13 Apr 2023 14:52:52 GMT

rex field=user mode=sed and then the expression?

Re: How to extract two fields from a group

ITWhisperer — Thu, 13 Apr 2023 14:54:41 GMT

No - mode=sed is for stream editing, which is not required when you are just extracting fields

Re: How to extract two fields from a group?

yuanliu — Fri, 14 Apr 2023 07:34:20 GMT

Is there any reason why you must use regex? For rigidly formatted strings like this, the easiest - in fact the cheapest solution is kv aka extract. Assuming your field name is log:

| rename _raw as temp, log as _raw | kv pairdelim=":" kvdelim="=" | rename _raw as log, temp as _raw

Your sample data should give you

cosId	fromhost	fromport	node	nodePort	respcode	resptime	sid
asasasa	999.99.99.99	3299	999999ss03	5002	OK	7	92ac3498-d95d-11ed-af19-92eb6037d638

Re: How to extract two fields from a group?

woodcock — Fri, 14 Apr 2023 18:29:39 GMT

Install the TA. It will do all of this.

Otherwise you can do this:
... | rex "fromhost=(?<fromhost>[^=:]+).*:cosId=(?<cosId>.*)$"

But it would be better to setup a sourcetype-based global extraction (which the TA surely does), like this:
(?<_KEY_1>[^=:]+)=(?<_VAL_1>[^=:]+)

Re: How to extract two fields from a group

kmhanson — Mon, 17 Apr 2023 11:35:51 GMT

I did play and get it to work. A big help. Thanks so much to both of you for the help.

Re: How to extract two fields from a group

dtsariapkin — Mon, 17 Apr 2023 11:41:14 GMT

@kmhanson Good to know. Don't forget to mark the answer as solution. And give people who helped Karma. That keeps them going!