Remove specific strings from raw events based on other fields

sandeepreddy947 — Mon, 20 Nov 2023 21:25:02 GMT

Firewall logs needs some purification for threat monitoring, below are couple events,

From the events below action=Accept AND Service=23 along with protection_type=geo_protection, we need "protection_type=geo_protection" to be removed from raw in indextime extraction.

Current:

2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513220|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=000|time=1700513220|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Denied|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=67|src=111.11.1.111|src_country=Other

Expected:

2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513220|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=000|time=1700513220|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|---|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|---|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Accept|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|---|proto=99|s_port=1234|service=23|src=111.11.1.111|src_country=Other 2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Denied|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=67|src=111.11.1.111|src_country=Other

Thanks in Advance!

Re: Remove specific strings from raw events based on other fields

richgalloway — Tue, 21 Nov 2023 13:58:07 GMT

Use SEDCMD in props.conf

[mysourcetype] SEDCMD-rm-geo_protection = s/protection_type=geo_protection/---/g

Re: Remove specific strings from raw events based on other fields

sandeepreddy947 — Tue, 21 Nov 2023 14:14:08 GMT

But, this regex with SED will replace in all events, i only need them replaced when action=Allowed and Service=23" in raw events. your regex will not satisfy below event.

2023-11-20K00:12:00-05:00 111.111.11.111 time=1700513221|hostname=firewallhost|product=Firewall|action=Denied|ifdir=inbound|ifname=eth3-01|logid=xxxx|loguid={xxxx,xxxx,xxxx,xxxx}|origin=111.111.11.111|originsicname=PK\=originsicname,O\=xpljdkdk..xpl78kdk|sequencenum=00|time=1700513221|version=5|dst=111.11.1.111|dst_country=PL|inspection_information=Geo-location outbound enforcement|inspection_profile=Geo_settings_upgraded_from_FWPRMLP_Internet_v4|protection_type=geo_protection|proto=99|s_port=1234|service=67|src=111.11.1.111|src_country=Other

Re: Remove specific strings from raw events based on other fields

richgalloway — Tue, 21 Nov 2023 15:25:55 GMT

Sorry about that. Try this SEDCMD, instead. It does, however, make some assumptions about the order of fields.

SEDCMD-rm-geo_protection = s/(.*\|action=Accept\|)(.*?)\|protection_type=geo_protection\|(.*?)(\|service=23.*)/\1\2|---|\3\4/

topic Re: Remove specific strings from raw events based on other fields in Splunk Search

Remove specific strings from raw events based on other fields

Re: Remove specific strings from raw events based on other fields

Re: Remove specific strings from raw events based on other fields

Re: Remove specific strings from raw events based on other fields