Hi Splunk Community,
I am seeking assistance on what should be a relatively simple task - to drop/filter particular events on a heavy forwarder node using props.conf/transforms.conf. I have successfully implemented filtering/dropping events within the environment many times before so I am surprised how how difficult to make work this particular transform/prop is. It is not working as expected despite the string matching regex working successfully when testing on the Splunk GUI, the Splunk pcregextest CLI tool and regex101.com.
The log events are from a Cisco Firepower firewall. They are sent via syslog from the firewall to a Linux syslog server where they are written to disk and then picked up by the SplunkUniversalForwarder, sent to a Heavy Forwarder node before being sent to an Index node.
I have tried many permutations of the props.conf/transforms.conf below including sending events which do NOT contain the specified words to the nullQueue or sending everything to the nullQueue then filtering events which DO match the regex to the indexQueue, various changes to the regex etc. the result is that the Splunk index node (and the resultant index) will either receive every event or no events at all (depending on the test).
Any help or tips to assist debugging this problem will be greatly appreciated.
Thanks,
What I am trying to achieve with props.conf/transforms.conf?
- Drop any log events which do not contain the word "URLSICategory" or "DNSSICategory" or "IPReputationSICategory"
inputs.conf (Linux syslog host - universal forwarder)
[monitor:///var/log/firewall/firewall_test.log]
disabled=false
index=ngfw_security_intelligence
sourcetype=security_syslog
transforms.conf (heavy forwarder)
[allsetnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[ngfw_si_events_whitelist]
REGEX = ((?:URLSICategory|DNSSICategory|IPReputationSICategory))
DEST_KEY = queue
FORMAT = indexQueue
props.conf (heavy forwarder)
[ngfw_security_intelligence]
TRANSFORMS-ngfw-drop = allsetnull, ngfw_si_events_whitelist
Other regex tried:
- Match lines which do NOT contain the words (then nullQueue them)
(?s)^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$
- Match lines which DO contain the words (then indexQueue them)
((?:URLSICategory|DNSSICategory|IPReputationSICategory))
(?:URLSICategory|DNSSICategory|IPReputationSICategory)
(URLSICategory|DNSSICategory|IPReputationSICategory)
URLSICategory|DNSSICategory|IPReputationSICategory
Testing done (GUI):
index=ngfw_security_intelligence | regex _raw="^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$"
index=ngfw_security_intelligence | regex _raw="(?s)^((?!URLSICategory:|DNSSICategory:|IPReputationSICategory:).)*$"
Testing done (CLI):
./splunk cmd pcregextest mregex="^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$" test_str="Destination, IPReputationSICategory: Global-Blacklist_1"
Testing done (regex101.com):
https://regex101.com/r/MDQqBx/1
https://regex101.com/r/rLDxHr/1
The log data (anonymized for this post):
Rcvd:2021-04-15T11:17:46.673993+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:45 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430002: EventPriority: High, DeviceUUID: 00000000-0000-0000-0000-000000000001, InstanceID: 16, FirstPacketSecond: 2021-04-15T01:17:45Z, ConnectionID: 35916, AccessControlRuleAction: Block, AccessControlRuleReason: IP Block, SrcIP: 10.0.0.8, DstIP: 1.2.3.9, SrcPort: 16403, DstPort: 16386, Protocol: udp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, Prefilter Policy: PFP-XX-20210329, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 58, ResponderBytes: 0, NAPPolicy: No Rules Active, SecIntMatchingIP: Destination, IPReputationSICategory: Global-Blacklist_1
Rcvd:2021-04-15T11:17:49.924536+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:49 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T01:17:49Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.1, DstIP: 1.2.3.4, SrcPort: 54102, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 12, ResponderPackets: 15, InitiatorBytes: 1419, ResponderBytes: 12575, NAPPolicy: No Rules Active
Rcvd:2021-04-15T12:53:40.111154+10:00 From:firewall-1.site.com Time:2021-04-15T02:53:39 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T02:53:39Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.2, DstIP: 1.2.3.5, SrcPort: 48012, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 10, ResponderPackets: 15, InitiatorBytes: 1678, ResponderBytes: 12575, NAPPolicy: No Rules Active
Rcvd:2021-04-15T12:53:40.112896+10:00 From:firewall-1.site.com Time:2021-04-15T02:53:40 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T02:53:40Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.3, DstIP: 1.2.3.6, SrcPort: 48019, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 11, ResponderPackets: 15, InitiatorBytes: 1678, ResponderBytes: 12575, NAPPolicy: No Rules Active
Rcvd:2021-04-15T12:53:40.123993+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:45 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430002: EventPriority: High, DeviceUUID: 00000000-0000-0000-0000-000000000002, InstanceID: 16, FirstPacketSecond: 2021-04-15T01:17:45Z, ConnectionID: 35916, AccessControlRuleAction: Block, AccessControlRuleReason: IP Block, SrcIP: 10.0.0.4, DstIP: 1.2.3.7, SrcPort: 16403, DstPort: 16386, Protocol: udp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, Prefilter Policy: PFP-XX-20210329, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 58, ResponderBytes: 0, NAPPolicy: No Rules Active, SecIntMatchingIP: Destination, IPReputationSICategory: Global-Blacklist_1
Rcvd:2021-04-15T12:53:40.130404+10:00 From:firewall-2.site.com Time:2021-04-15T12:53:40 Host:firewall-2 Pri:local4.err Msg::Apr 15 02:53:40 UTC: %FTD-session-3-106014: Deny inbound icmp src Inside:10.1.0.1 dst nlp_int_tap:169.254.1.2 (type 3, code 3)
I thought I would comment and provide the solution for this in case other community users run into a similar problem. I was able to get this working by specifying source::<source> in props.conf rather than the index name. Index names are not supported in props.conf, the existing entries we had which I thought were operating on index name were actually operating on the sourcetype which had been set on the Universal Forwarder client in inputs.conf with a name similar to the index name.
https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Propsconf
<spec> can be: 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host, or host-matching pattern, for an event. 3. source::<source>, where <source> is the source, or source-matching pattern, for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen.
The correct entry for props.conf is:
[source::/var/log/firewall/firewall_test.log]
TRANSFORMS-ngfw-drop-non-si-events = allsetnull, ngfw_si_events_whitelist
The correct entries for transforms.conf are:
[allsetnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[ngfw_si_events_whitelist]
REGEX = (URLSICategory|DNSSICategory|IPReputationSICategory)
DEST_KEY = queue
FORMAT = indexQueue
I thought I would comment and provide the solution for this in case other community users run into a similar problem. I was able to get this working by specifying source::<source> in props.conf rather than the index name. Index names are not supported in props.conf, the existing entries we had which I thought were operating on index name were actually operating on the sourcetype which had been set on the Universal Forwarder client in inputs.conf with a name similar to the index name.
https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Propsconf
<spec> can be: 1. <sourcetype>, the source type of an event. 2. host::<host>, where <host> is the host, or host-matching pattern, for an event. 3. source::<source>, where <source> is the source, or source-matching pattern, for an event. 4. rule::<rulename>, where <rulename> is a unique name of a source type classification rule. 5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed source type classification rule. These are only considered as a last resort before generating a new source type based on the source seen.
The correct entry for props.conf is:
[source::/var/log/firewall/firewall_test.log]
TRANSFORMS-ngfw-drop-non-si-events = allsetnull, ngfw_si_events_whitelist
The correct entries for transforms.conf are:
[allsetnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[ngfw_si_events_whitelist]
REGEX = (URLSICategory|DNSSICategory|IPReputationSICategory)
DEST_KEY = queue
FORMAT = indexQueue
Hi @markpdeakin
maybe is a stupid question but why do you have two regex on your transforms stanza?
[ngfw_si_events_whitelist]
REGEX = ((?:URLSICategory|DNSSICategory|IPReputationSICategory))
REGEX = IPReputationSICategory
DEST_KEY = queue
FORMAT = indexQueue
Couldn't he work with just one? maybe the first
Hi aasabatini,
Thanks for picking up on this error, I have update the original post to only include a single regex line. The second regex line was actually #commented out on my Splunk heavy forwarder system and I messed up the formatting when copying the configs across to this post.
Thanks,