topic Re: Extract multiple IP addresses from _raw and assign same field name in Getting Data In

Extract multiple IP addresses from _raw and assign same field name

kaw243 — Tue, 29 Sep 2020 13:48:13 GMT

I am trying to extract all IP addresses from _raw with a field name of rf_ip so that I can use this value to do a lookup for any IP in the logs that match, but I seem to have something configured incorrectly.

I have tested the regex in regex101 and within splunk using:

| rex field=_raw "(?<"rf_ip">\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b)" max_match=0

(I have "" around <"rf_ip"> so it will show up in the forum)

Here are my configs:
On HFs

transforms.conf

[rf_ip]

1. REGEX = (?<"rf_ip">\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b)
(I have "" around <"rf_ip"> so it will show up in the forum)
2. SOURCE_KEY = _raw
3. MV_ADD = true
4. CAN_OPTIMIZE = false

props.conf

[infoblox:dns]
TRANSFORMS-rfip = rf_ip

ES search head /etc/system/local
fields.conf

[rf_ip]
INDEXED = True

Splunk design:
Universal forwarders or syslog => HF => indexers => ES search head

Re: Extract multiple IP addresses from _raw and assign same field name

richgalloway — Tue, 25 Apr 2017 15:10:11 GMT

The quotation marks around field names aren't needed if you highlight that text and click the '101010' button.

Re: Extract multiple IP addresses from _raw and assign same field name

beatus — Tue, 25 Apr 2017 16:54:25 GMT

A couple of issues here:

Don't use indexed fields for this. Unless you have a very specific reason for trying to use an indexed field you're just causing more issues than you're solving.
Because of your use of indexed fields, the MV_ADD does not work.

A fix is going to be using search time fields like so:
props.conf:

[rf_ip]
REPORT_rfip = rf_ip

transforms.conf:

[rf_ip]
REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b
MV_ADD = true

Lastly, remove your fields.conf. All of this needs to exist on the Search Head, as it's a search time configuration.

Re: Extract multiple IP addresses from _raw and assign same field name

kaw243 — Tue, 29 Sep 2020 13:48:21 GMT

I removed the configs from the HF as well as the fields.conf from my search head. I also added the above (without the "" around rf_ip) to my search head in /etc/system/local and restarted it, but I am still not seeing the rf_ip field extracted.

Re: Extract multiple IP addresses from _raw and assign same field name

beatus — Tue, 25 Apr 2017 18:31:30 GMT

Can you share some sample data?

Re: Extract multiple IP addresses from _raw and assign same field name

kaw243 — Tue, 29 Sep 2020 13:48:24 GMT

Here are a few sample logs.

Apr 25 13:40:06 aws1oainblx001.xxx.corp 12.123.32.50 named[669]: 25-Apr-2017 13:40:06.365 client 112.128.532.20#3876: view 3: UDP: query: assets.locuslabs.com IN A response: NOERROR + assets.locuslabs.com. 60 IN A 53.192.139.36; assets.locuslabs.com. 60 IN A 51.2.39.84; assets.locuslabs.com. 60 IN A 54.142.13.2; assets.locuslabs.com. 60 IN A 52.19.9.65; assets.locuslabs.com. 60 IN A 54.122.39.11; assets.locuslabs.com. 60 IN A 53.92.1.101; assets.locuslabs.com. 60 IN A 54.112.19.0; assets.locuslabs.com. 60 IN A 54.12.19.18;

Apr 25 13:40:06 aws1dsfdn102.xxx.corp 134.45.88.163 named[9622]: 25-Apr-2017 13:40:06.994 client 140.44.112.233#3470: UDP: query: localhost.nao.xxx.com IN A response: NXDOMAIN +

time=1493145885|loc=4388995|fileid=1493096349|action=drop|orig=xxx_FW|i/f_dir=inbound|i/f_name=eth1-04|has_accounting=0|uuid=<00000000,00000000,00000000,00000000>|product=VPN-1 & FireWall-1|TCP packet out of state=First packet isn't SYN|tcp_flags=FIN-ACK|src=h-132.129.113.177|s_port=24543|dst=129.65.119.203|service=http|proto=tcp|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={7460A7102-C6476-A4E43-9C37A-E870A497F5537};mgmt=xxx_cma;date=1492733045;policy_name=xxx]|origin_sic_name=CN=xxx_FW,O=xxx_cma..xxx

Apr 25 13:49:14 aws.xxx.corp 110.40.32.50 named[669]: 25-Apr-2017 13:49:14.898 client 122.168.52.210#62481: view 3: UDP: query: guzzoni.apple.com IN A response: NOERROR + guzzoni.apple.com. 120 IN CNAME origin.guzzoni-apple.com.akadns.net.; origin.guzzoni-apple.com.akadns.net. 241 IN CNAME st14p01sa.guzzoni-apple.com.akadns.net.; st14p01sa.guzzoni-apple.com.akadns.net. 298 IN A 127.174.1.5;

Apr 25 13:49:14 xxx.xxx.corp 120.40.32.50 named[669]: client 112.168.52.210#62481 (guzzoni.apple.com): view 3: query: guzzoni.apple.com IN A + (120.40.32.50)

Apr 25 13:49:14 xxx.xxx.corp 130.40.32.50 named[669]: 25-Apr-2017 13:49:14.894 client 192.168.52.210#63019: view 3: UDP: query: apple.com IN A response: NOERROR + apple.com. 1350 IN A 197.172.224.47; apple.com. 1350 IN A 147.178.96.59; apple.com. 1350 IN A 170.142.160.59;

Apr 25 13:49:14 xxx.xxx.corp 170.40.32.50 named[669]: client 190.168.52.210#63019 (apple.com): view 3: query: apple.com IN A + (134.140.32.50)

Re: Extract multiple IP addresses from _raw and assign same field name

beatus — Wed, 26 Apr 2017 13:33:38 GMT

Okay, lets change our regular expression to match this a bit better then:

Props.conf:

[rf_ip]
REPORT_rfip = rf_ip

Transforms.conf:

[rf_ip]
REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b
MV_ADD = true

Here's the result on my box:

Re: Extract multiple IP addresses from _raw and assign same field name

kaw243 — Tue, 29 Sep 2020 13:48:58 GMT

Thanks beatus for your help with this, it was greatly appreciated.

The final config that worked for me is
props.conf
[(?::){0}*] <== used this to use for all source type
REPORT-rf_ip = rf_ip <== updated this from REPORT_rf to REPORT-rf

transforms.conf
[rf_ip]
REGEX = \b(?\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b
MV_ADD = true