topic Re: Using Regex to split a field in Splunk Search

Using Regex to split a field?

anandhalagaras1 — Thu, 19 Jan 2023 15:33:45 GMT

Hi Team,

I have sample set of events coming from the same logs and here "x" denotes a digit mostly IP address in this case and my requirement is that to split the data in the existing field "Forwarder" which is mentioned as "v". So already we have a field extraction in place i.e. the name of field is "Forwarder". And the current output is as below from all the 5 sample events and for the 5th sample event we don't have the "vvv" value itself in the logs.

Sample Logs:

2021-02-12 06:23:17 xx.xxx.xxx.xx GET /test/v1/xyz/abc/domainsetting domainName=xx.xxx.xxx.xx 443 - xx.xxx.x.xxx function/xxx.x.x.x - xxx x x xx vv.vvv.vvv.vv

2021-02-12 06:23:26 xx.xxx.xxx.xx GET /Window-2020-def-yy-ab - 443 - xx.xxx.x.xxx Mozilla/5.0+(compatible;+MSIE+9.0;+Windows+NT+6.1;+Linuxx/5.0;+AppInfo) - xxx x x xxx vv.vv.v.v

2021-02-12 06:11:55 xx.xxx.xxx.xx POST /test/abc/api/Control/Match - 443 - xx.xxx.x.xxx Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/88.0.4324.150+Safari/537.36 https://abc.def-mm.com/abc/def/dashboard/DeliveryList/DeliveryDetail?deliveryId=xxxxx&deliverySource=Feed xxx x x xxx vvv.vv.vvv.vvv,+vv.vvv.v.vvv,+vv.vvv.vvv.vv

2021-02-12 01:14:47 xx.xxx.xxx.xx GET /test/Abcdefgh/login+button+with+xyz.jpg - 443 - xx.xxx.x.xxx Mozilla/5.0+(iPhone;+CPU+iPhone+OS+14_4+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Mobile/xxxxx - xxx x x x vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv,+vvv.vvv.vvv.vvv

2021-02-12 07:32:20 xx.xxx.xxx.xx GET / - 443 - xx.xxx.x.xx - - x xx x x -

Forwarder (field name)

vv.vvv.vvv.vv
vv.vv.v.v
vvv.vv.vvv.vvv,+vv.vvv.v.vvv,+vv.vvv.vvv.vv
vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv,+vvv.vvv.vvv.vvv

So I want to split them up in the same field name "Forwarder" i.e. Consider the 3rd (vvv.vv.vvv.vvv,+vv.vvv.v.vvv,+vv.vvv.vvv.vv) & 4th output (vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv:vvvv,+vvv.vvv.vvv.vvv) in which the "Forwarder" field has multiple IP's in the same field for the 3rd and 4th output so we want to extract in separate fields under "Forwarder" with each IP.

Eg:
Forwarder (field name)
123.456.78.912,+45.675.3.123,+32.123.456.55

Output should be as below for Forwarder Field name as:
123.456.78.912
45.675.3.123
32.123.456.55

So kindly help me with the regex for the same.

Re: Using Regex to split a field

saravanan90 — Fri, 12 Feb 2021 07:57:48 GMT

This may help...

|makeresults | eval forwarder="123.456.78.912,+45.675.3.123,+32.123.456.55" | makemv delim="," forwarder

(or)

|makeresults | eval forwarder="123.456.78.912,+45.675.3.123,+32.123.456.55" | rex field=forwarder max_match=0 "(?<Forwarder>[^\,]+)"

Re: Using Regex to split a field

anandhalagaras1 — Fri, 12 Feb 2021 14:30:53 GMT

Thank you for your response.

But in your search query you have provided the example IP which I have mentioned but in similar type we have so many IP's and moreover I want to use the command and sort the details as well in the query.

index=abc sourcetype=xyz* | stats count by Forwarder | sort -count

So i want to split the data and their count as well. So we can work on our exact requirement. So kindly help on the query.

Re: Using Regex to split a field

saravanan90 — Fri, 12 Feb 2021 15:08:50 GMT

Please check if the below query helps.

Re: Using Regex to split a field

anandhalagaras1 — Fri, 12 Feb 2021 16:01:16 GMT

@saravanan90

Thank you for your response.

So instead of mentioning all the IP's in eval Forwarder part in the query can we mention something like * since there are multiple number of IP's so we cant able to mention all of them. Also for another set of sourcetype we have the Forwarder field extracted as well. So can we include the index and sourcetype as well in the same query so that it will be unique as well. Hence Kindly help on it.

Re: Using Regex to split a field

saravanan90 — Fri, 12 Feb 2021 16:16:52 GMT

Yes. We need to add the base search with index, sourcetype & other filters to extract the events and get the forwarder ip value as mentioned.

Eg:
Forwarder (field name)
123.456.78.912,+45.675.3.123,+32.123.456.55

Re: Using Regex to split a field

anandhalagaras1 — Mon, 15 Feb 2021 14:27:26 GMT

Hi Saravanan,

Thank you for your response.

But in few cases we have more than 30+ IP's so for each and everything how can we enter the IP in the Forwarder field to search the logs instead can we use something like * i.e. It can be whatever might be the IP but it needs to be arranged properly in the Forwarder field.

Example:

123.456.789

123.456.789,+321.345.456.432,+987.654.321.123,+875.453.23.345,+.......

So I want the output to be as below :

Forwarder Count

123.456.789 2

321.345.456.432 1

987.654.321.123 1

875.453.23.345 1

And also the count is restricted to 10 but we want all the values split up as above. So kindly help with the query with split up along with count.

Re: Using Regex to split a field

anandhalagaras1 — Wed, 17 Feb 2021 07:51:16 GMT

Can anyone help me regarding my requirement with field split up.

Re: Using Regex to split a field

scelikok — Wed, 17 Feb 2021 13:26:34 GMT

Hi @anandhalagaras1,

@saravanan90 solution should be working but I think there is misunderstanding with the sample eval. You don't need to write your ip addresses, they should come from your search already.

In your question you told us Forwarder field has ip addresses, please try adding below in your search.

| makemv Forwarder delim="," | mvexpand Forwarder | eval Forwarder=replace(Forwarder, "\+", "") | stats count by Forwarder

Re: Using Regex to split a field

anandhalagaras1 — Wed, 17 Feb 2021 16:06:47 GMT

@scelikok ,

Thanks for your response.

So now I have ran the query as you have mentioned:

index=abc sourcetype=def
| makemv Forwarder delim=","
| mvexpand Forwarder
| eval Forwarder=replace(Forwarder, "\+", "")
| stats count by Forwarder

And after which the first field is blank and I can see huge number of count and for the rest of the field I can see IP's split up with count.

So why the first field is blank with no information has so much of count whereas the rest has the IP and count.

Sample output:

Forwarder Count

4500

123.456.78.432 23

345.342.543.123 12

Re: Using Regex to split a field

scelikok — Wed, 17 Feb 2021 17:48:47 GMT

You may have Forwarder field with empty value. You can filter them before makemv command;

Re: Using Regex to split a field

anandhalagaras1 — Fri, 19 Feb 2021 11:52:38 GMT

@scelikok ,

Thank you it worked like a charm.

Re: Using Regex to split a field

karthikvj — Thu, 19 Jan 2023 14:49:44 GMT

If you could get the Forwarder field contains value: "123.456.78.912,+45.675.3.123,+32.123.456.55"

| rex field=forwarder "(?<numbers>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})" max_match=3

max_match parameter should be maximum number since forwarder might have more than 10 values, depending on your use case.