topic Re: How can I replace only the field value if found using Automatic lookups? in Splunk Search

How can I replace only the field value if found using Automatic lookups?

horst_poehlmann — Mon, 28 Sep 2020 18:04:57 GMT

I have a problem with my checkpoint logs and automatic lookup tables (although the problem is not specific to checkpoint logs but rather automatic lookups).

The original log lines look something like this:

loc=1959996|time= 6Nov2014 11:48:49|action=accept|orig=fw|i/f_dir=inbound|i/f_name=Lan1|has_accounting=1|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={999999999-63E0-11E4-8888-999999999999};mgmt=cpmgr;date=1415077659;policy_name=POL1]|service_id=domain-udp|src=SERVER1|s_port=64444|dst=DESTSERV1|service=domain-udp|proto=udp|rule=77

loc=1960000|time= 6Nov2014 11:48:49|action=accept|orig=fw|i/f_dir=inbound|i/f_name=Lan2|has_accounting=1|product=VPN-1 & FireWall-1|__policy_id_tag=product=VPN-1 & FireWall-1[db_tag={999999999-63E0-11E4-8888-999999999999};mgmt=cpmgr;date=1415077659;policy_name=POL1]|service_id=https|src=10.0.0.1|s_port=55555|dst=10.1.0.1|service=https|proto=tcp|rule=222

When splunk creates the fields, I end up with src=SERVER1 and src_ip=SERVER1 (same for dest and dest_ip)

What I want to do is replace src_ip with the actual IP of SERVER1, so I created a csv from the checkpoint config (using CPrules.pl) and that all works fine.

However, there are also lines that are not defined and therefore already contain an IP address (see logline 2 above), which means it can't find it in the lookup table and therefore returns a blank field. I use the "Overwrite field values" feature in automatic lookups.

My Question is, can I somehow keep the original value if it wasn't found in the lookup table?

Also, when defining lookups in the Web GUI, where is the information stored?

PS: I use splunk 6.1

Thanks

Re: How can I replace only the field value if found using Automatic lookups?

musskopf — Mon, 28 Sep 2020 18:04:59 GMT

Hello horst.poehlmann,

All configuration is stored inside $SPLUNK_HOME/etc//local on the files props.conf and transforms.conf when permissions are "App" or "Global".

Related to your query, I believe the best thing is to output the IP from the automatic lookup to a new field, something like: src_ip_resolved and use an eval command to define it the src_ip should be replaced with the content of src_ip_resolved or not:

Example

<your search> | eval src_ip=IF(ISNULL(src_ip_resolved),src_ip,src_ip_resolved)

On the example above, you initial search should use the automatic lookup definition returning the new field src_ip_resolved, later you check: IF the lookup result was null, you use the src_ip, ELSE use src_ip_resolved instead. You'll need to repeat it for the dest_ip field as well.

Re: How can I replace only the field value if found using Automatic lookups?

horst_poehlmann — Mon, 10 Nov 2014 03:52:28 GMT

I could not find any of the variables I used in my lookup definitions using the GUI in the files under $SPLUNK_HOME. I grep'd for a particular string. Not that important at this stage though.

Thanks, the eval function did the trick. Not sure how efficient that is as I had to use 4 separate eval functions (src, dest port and protocol), but it does work.

Appreciate your answer.

Re: How can I replace only the field value if found using Automatic lookups?

musskopf — Mon, 10 Nov 2014 03:55:20 GMT

That's good! Eval functions like those ones, with a simple IF are very light... you can use the "JOB -> Inspect" panel to have a look how much time they take to run.