I am trying to extract all IP addresses from _raw with a field name of rf_ip so that I can use this value to do a lookup for any IP in the logs that match, but I seem to have something configured incorrectly.
I have tested the regex in regex101 and within splunk using:
| rex field=_raw "(?<"rf_ip">\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b)" max_match=0
(I have "" around <"rf_ip"> so it will show up in the forum)
Here are my configs:
On HFs
transforms.conf
[rf_ip]
1. REGEX = (?<"rf_ip">\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b)
(I have "" around <"rf_ip"> so it will show up in the forum)
2. SOURCE_KEY = _raw
3. MV_ADD = true
4. CAN_OPTIMIZE = false
props.conf
[infoblox:dns]
TRANSFORMS-rfip = rf_ip
ES search head /etc/system/local
fields.conf
[rf_ip]
INDEXED = True
Splunk design:
Universal forwarders or syslog => HF => indexers => ES search head
A couple of issues here:
A fix is going to be using search time fields like so:
props.conf:
[rf_ip]
REPORT_rfip = rf_ip
transforms.conf:
[rf_ip]
REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b
MV_ADD = true
Lastly, remove your fields.conf. All of this needs to exist on the Search Head, as it's a search time configuration.
A couple of issues here:
A fix is going to be using search time fields like so:
props.conf:
[rf_ip]
REPORT_rfip = rf_ip
transforms.conf:
[rf_ip]
REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b
MV_ADD = true
Lastly, remove your fields.conf. All of this needs to exist on the Search Head, as it's a search time configuration.
I removed the configs from the HF as well as the fields.conf from my search head. I also added the above (without the "" around rf_ip) to my search head in /etc/system/local and restarted it, but I am still not seeing the rf_ip field extracted.
Can you share some sample data?
Here are a few sample logs.
Apr 25 13:40:06 aws1oainblx001.xxx.corp 12.123.32.50 named[669]: 25-Apr-2017 13:40:06.365 client 112.128.532.20#3876: view 3: UDP: query: assets.locuslabs.com IN A response: NOERROR + assets.locuslabs.com. 60 IN A 53.192.139.36; assets.locuslabs.com. 60 IN A 51.2.39.84; assets.locuslabs.com. 60 IN A 54.142.13.2; assets.locuslabs.com. 60 IN A 52.19.9.65; assets.locuslabs.com. 60 IN A 54.122.39.11; assets.locuslabs.com. 60 IN A 53.92.1.101; assets.locuslabs.com. 60 IN A 54.112.19.0; assets.locuslabs.com. 60 IN A 54.12.19.18;
Apr 25 13:40:06 aws1dsfdn102.xxx.corp 134.45.88.163 named[9622]: 25-Apr-2017 13:40:06.994 client 140.44.112.233#3470: UDP: query: localhost.nao.xxx.com IN A response: NXDOMAIN +
time=1493145885|loc=4388995|fileid=1493096349|action=drop|orig=xxx_FW|i/f_dir=inbound|i/f_name=eth1-04|has_accounting=0|uuid=<00000000,00000000,00000000,00000000>|product=VPN-1 & FireWall-1|TCP packet out of state=First packet isn't SYN|tcp_flags=FIN-ACK|src=h-132.129.113.177|s_port=24543|dst=129.65.119.203|service=http|proto=tcp|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={7460A7102-C6476-A4E43-9C37A-E870A497F5537};mgmt=xxx_cma;date=1492733045;policy_name=xxx]|origin_sic_name=CN=xxx_FW,O=xxx_cma..xxx
Apr 25 13:49:14 aws.xxx.corp 110.40.32.50 named[669]: 25-Apr-2017 13:49:14.898 client 122.168.52.210#62481: view 3: UDP: query: guzzoni.apple.com IN A response: NOERROR + guzzoni.apple.com. 120 IN CNAME origin.guzzoni-apple.com.akadns.net.; origin.guzzoni-apple.com.akadns.net. 241 IN CNAME st14p01sa.guzzoni-apple.com.akadns.net.; st14p01sa.guzzoni-apple.com.akadns.net. 298 IN A 127.174.1.5;
Apr 25 13:49:14 xxx.xxx.corp 120.40.32.50 named[669]: client 112.168.52.210#62481 (guzzoni.apple.com): view 3: query: guzzoni.apple.com IN A + (120.40.32.50)
Apr 25 13:49:14 xxx.xxx.corp 130.40.32.50 named[669]: 25-Apr-2017 13:49:14.894 client 192.168.52.210#63019: view 3: UDP: query: apple.com IN A response: NOERROR + apple.com. 1350 IN A 197.172.224.47; apple.com. 1350 IN A 147.178.96.59; apple.com. 1350 IN A 170.142.160.59;
Apr 25 13:49:14 xxx.xxx.corp 170.40.32.50 named[669]: client 190.168.52.210#63019 (apple.com): view 3: query: apple.com IN A + (134.140.32.50)
Okay, lets change our regular expression to match this a bit better then:
Props.conf:
[rf_ip]
REPORT_rfip = rf_ip
Transforms.conf:
[rf_ip]
REGEX = \b(?<rf_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b
MV_ADD = true
Here's the result on my box:
Thanks beatus for your help with this, it was greatly appreciated.
The final config that worked for me is
props.conf
[(?::){0}*] <== used this to use for all source type
REPORT-rf_ip = rf_ip <== updated this from REPORT_rf to REPORT-rf
transforms.conf
[rf_ip]
REGEX = \b(?\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b
MV_ADD = true
The quotation marks around field names aren't needed if you highlight that text and click the '101010' button.