Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About markpdeakin
markpdeakin
Explorer
Member since:
04-26-2021
04-29-2021
Community Statistics
| Posts | 3 |
| Solutions | 1 |
| Karma Given | 1 |
| Karma Received | 0 |
| Member Since | 04-26-2021 |
Activity Feed
- Posted Re: Unable to drop/filter certain events on Heavy Forwarder using props.conf and transform.conf on Splunk Enterprise. 04-29-2021 06:13 PM
- Tagged Re: Unable to drop/filter certain events on Heavy Forwarder using props.conf and transform.conf on Splunk Enterprise. 04-29-2021 06:13 PM
- Karma Re: Unable to drop/filter certain events on Heavy Forwarder using props.conf and transform.conf for aasabatini. 04-27-2021 04:14 PM
- Posted Re: Unable to drop/filter certain events on Heavy Forwarder using props.conf and transform.conf on Splunk Enterprise. 04-27-2021 03:18 PM
- Posted Unable to drop/filter certain events on Heavy Forwarder using props.conf and transform.conf on Splunk Enterprise. 04-26-2021 06:56 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 |
04-29-2021
06:13 PM
I thought I would comment and provide the solution for this in case other community users run into a similar problem. I was able to get this working by specifying source::<source> in props.conf rather than the index name. Index names are not supported in props.conf, the existing entries we had which I thought were operating on index name were actually operating on the sourcetype which had been set on the Universal Forwarder client in inputs.conf with a name similar to the index name. https://docs.splunk.com/Documentation/Splunk/Latest/Admin/Propsconf <spec> can be:
1. <sourcetype>, the source type of an event.
2. host::<host>, where <host> is the host, or host-matching pattern, for an
event.
3. source::<source>, where <source> is the source, or source-matching
pattern, for an event.
4. rule::<rulename>, where <rulename> is a unique name of a source type
classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed
source type classification rule.
These are only considered as a last resort
before generating a new source type based on the
source seen. The correct entry for props.conf is: [source::/var/log/firewall/firewall_test.log] TRANSFORMS-ngfw-drop-non-si-events = allsetnull, ngfw_si_events_whitelist The correct entries for transforms.conf are: [allsetnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [ngfw_si_events_whitelist] REGEX = (URLSICategory|DNSSICategory|IPReputationSICategory) DEST_KEY = queue FORMAT = indexQueue
... View more
- Tags:
- Solved
04-27-2021
03:18 PM
Hi aasabatini , Thanks for picking up on this error, I have update the original post to only include a single regex line. The second regex line was actually #commented out on my Splunk heavy forwarder system and I messed up the formatting when copying the configs across to this post. Thanks,
... View more
04-26-2021
06:56 PM
Hi Splunk Community, I am seeking assistance on what should be a relatively simple task - to drop/filter particular events on a heavy forwarder node using props.conf/transforms.conf. I have successfully implemented filtering/dropping events within the environment many times before so I am surprised how how difficult to make work this particular transform/prop is. It is not working as expected despite the string matching regex working successfully when testing on the Splunk GUI, the Splunk pcregextest CLI tool and regex101.com. The log events are from a Cisco Firepower firewall. They are sent via syslog from the firewall to a Linux syslog server where they are written to disk and then picked up by the SplunkUniversalForwarder, sent to a Heavy Forwarder node before being sent to an Index node. I have tried many permutations of the props.conf/transforms.conf below including sending events which do NOT contain the specified words to the nullQueue or sending everything to the nullQueue then filtering events which DO match the regex to the indexQueue, various changes to the regex etc. the result is that the Splunk index node (and the resultant index) will either receive every event or no events at all (depending on the test). Any help or tips to assist debugging this problem will be greatly appreciated. Thanks, What I am trying to achieve with props.conf/transforms.conf? - Drop any log events which do not contain the word "URLSICategory" or "DNSSICategory" or "IPReputationSICategory" inputs.conf (Linux syslog host - universal forwarder) [monitor:///var/log/firewall/firewall_test.log] disabled=false index=ngfw_security_intelligence sourcetype=security_syslog transforms.conf (heavy forwarder) [allsetnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [ngfw_si_events_whitelist] REGEX = ((?:URLSICategory|DNSSICategory|IPReputationSICategory)) DEST_KEY = queue FORMAT = indexQueue props.conf (heavy forwarder) [ngfw_security_intelligence] TRANSFORMS-ngfw-drop = allsetnull, ngfw_si_events_whitelist Other regex tried: - Match lines which do NOT contain the words (then nullQueue them) (?s)^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$ - Match lines which DO contain the words (then indexQueue them) ((?:URLSICategory|DNSSICategory|IPReputationSICategory)) (?:URLSICategory|DNSSICategory|IPReputationSICategory) (URLSICategory|DNSSICategory|IPReputationSICategory) URLSICategory|DNSSICategory|IPReputationSICategory Testing done (GUI): index=ngfw_security_intelligence | regex _raw="^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$" index=ngfw_security_intelligence | regex _raw="(?s)^((?!URLSICategory:|DNSSICategory:|IPReputationSICategory:).)*$" Testing done (CLI): ./splunk cmd pcregextest mregex="^((?!URLSICategory|DNSSICategory|IPReputationSICategory).)*$" test_str="Destination, IPReputationSICategory: Global-Blacklist_1" Testing done (regex101.com): https://regex101.com/r/MDQqBx/1 https://regex101.com/r/rLDxHr/1 The log data (anonymized for this post): Rcvd:2021-04-15T11:17:46.673993+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:45 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430002: EventPriority: High, DeviceUUID: 00000000-0000-0000-0000-000000000001, InstanceID: 16, FirstPacketSecond: 2021-04-15T01:17:45Z, ConnectionID: 35916, AccessControlRuleAction: Block, AccessControlRuleReason: IP Block, SrcIP: 10.0.0.8, DstIP: 1.2.3.9, SrcPort: 16403, DstPort: 16386, Protocol: udp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, Prefilter Policy: PFP-XX-20210329, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 58, ResponderBytes: 0, NAPPolicy: No Rules Active, SecIntMatchingIP: Destination, IPReputationSICategory: Global-Blacklist_1 Rcvd:2021-04-15T11:17:49.924536+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:49 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T01:17:49Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.1, DstIP: 1.2.3.4, SrcPort: 54102, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 12, ResponderPackets: 15, InitiatorBytes: 1419, ResponderBytes: 12575, NAPPolicy: No Rules Active Rcvd:2021-04-15T12:53:40.111154+10:00 From:firewall-1.site.com Time:2021-04-15T02:53:39 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T02:53:39Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.2, DstIP: 1.2.3.5, SrcPort: 48012, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 10, ResponderPackets: 15, InitiatorBytes: 1678, ResponderBytes: 12575, NAPPolicy: No Rules Active Rcvd:2021-04-15T12:53:40.112896+10:00 From:firewall-1.site.com Time:2021-04-15T02:53:40 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430003: EventPriority: Low, DeviceUUID: 00000000-0000-0000-0000-000000000001, FirstPacketSecond: 2021-04-15T02:53:40Z, ConnectionID: 0, AccessControlRuleAction: Allow, SrcIP: 10.0.0.3, DstIP: 1.2.3.6, SrcPort: 48019, DstPort: 443, Protocol: tcp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, AccessControlRuleName: From-IPv4-allowed-users, Prefilter Policy: PFP-XX-20210329, User: Not Found, ConnectionDuration: 0, InitiatorPackets: 11, ResponderPackets: 15, InitiatorBytes: 1678, ResponderBytes: 12575, NAPPolicy: No Rules Active Rcvd:2021-04-15T12:53:40.123993+10:00 From:firewall-1.site.com Time:2021-04-15T01:17:45 Host:firewall-1 Pri:alert.info Msg: %FTD-6-430002: EventPriority: High, DeviceUUID: 00000000-0000-0000-0000-000000000002, InstanceID: 16, FirstPacketSecond: 2021-04-15T01:17:45Z, ConnectionID: 35916, AccessControlRuleAction: Block, AccessControlRuleReason: IP Block, SrcIP: 10.0.0.4, DstIP: 1.2.3.7, SrcPort: 16403, DstPort: 16386, Protocol: udp, IngressInterface: Inside, EgressInterface: Outside, IngressZone: INSIDE, EgressZone: OUTSIDE, IngressVRF: Global, EgressVRF: Global, ACPolicy: ACP-XX-20210329, Prefilter Policy: PFP-XX-20210329, InitiatorPackets: 1, ResponderPackets: 0, InitiatorBytes: 58, ResponderBytes: 0, NAPPolicy: No Rules Active, SecIntMatchingIP: Destination, IPReputationSICategory: Global-Blacklist_1 Rcvd:2021-04-15T12:53:40.130404+10:00 From:firewall-2.site.com Time:2021-04-15T12:53:40 Host:firewall-2 Pri:local4.err Msg::Apr 15 02:53:40 UTC: %FTD-session-3-106014: Deny inbound icmp src Inside:10.1.0.1 dst nlp_int_tap:169.254.1.2 (type 3, code 3)
... View more
Labels
Contact Me
| Online Status |
Offline
|
| Date Last Visited |
04-29-2021
09:45 PM
|
