Splunk log filtering via Splunk Heavy forwarder

[Sayanta_Basak_I]

Hello,

I have the the logs coming from our IPS coming in below format:

Connection logs:
Aug 10 08:54:44 HOST SFIMS: Protocol: TCP, SrcIP: 10.1.1.1, OriginalClientIP: ::, DstIP: 10.2.2., SrcPort: 58457, DstPort: 80, TCPFlags: 0x0, IngressInterface: Default-VRF, EgressInterface: Prod, IngressZone: Default-VRF-location, EgressZone: Prod-location DE: Primary Detection Engine (7eaa2610-2c9a-11e7-a3ec-f9112474357f), Policy: LO-Access-Policy, ConnectType: End, AccessControlRuleName: DefaultVRF to DCLAN, AccessControlRuleAction: Allow, Prefilter Policy: Unknown, UserName: No Authentication Required, InitiatorPackets: 3, ResponderPackets: 2, InitiatorBytes: 234, ResponderBytes: 136, NAPPolicy: Balanced Security and Connectivity, DNSResponseType: No Error, Sinkhole: Unknown, URLCategory: Unknown, URLReputation: Risk unknown

If I want to filter the events and pick only below information via a Heavy Forwarder will Splunk allow me to do that?

Connection logs:
Aug 10 08:54:44 HOST SFIMS: Protocol: TCP, SrcIP: 10.1.1.1, OriginalClientIP: ::, DstIP: 10.2.2., SrcPort: 58457, DstPort: 80, TCPFlags: 0x0, IngressInterface: Default-VRF, EgressInterface: Prod, ConnectType: End, AccessControlRuleAction: Allow, UserName: No Authentication Required

[s2_splunk]

Side comment: If you don't already have a heavy forwarder in your environment, don't introduce one just to do the filtering unless you ensure you are not creating a bottleneck.
You can filter unwanted things on the indexer directly and filtered stuff does not count against your license.

[mdsnmss]

Use props and transforms on the heavy forwarder.

Props.conf

[sourcetype/source/etc]
TRANSFORM-trim_event = trim_event

Transforms.conf

[trim_event]
DEST_KEY = _raw
REGEX = (.*UserName: .+?),
FORMAT = $1

You might have to adjust the regex based on your needs. I am not a regex wizard but base on your sample event that would capture what you want.

[Sayanta_Basak_I]

Hello @mdsnmss

Thank you for your response. I am trying to understand how this might work before implementing it. Since your regex just includes username, does this mean only username info will be retained and I have to make similar comma separated inputs to onboard other infos. My requirement is to convert the source log dump to something limited as below

Connection logs:
Aug 10 08:54:44 HOST SFIMS: Protocol: TCP, SrcIP: 10.1.1.1, OriginalClientIP: ::, DstIP: 10.2.2., SrcPort: 58457, DstPort: 80, TCPFlags: 0x0, IngressInterface: Default-VRF, EgressInterface: Prod, ConnectType: End, AccessControlRuleAction: Allow, UserName: No Authentication Required

[mdsnmss]

The way the regex works is it will capture everything between (). You can test by using something like https://regex101.com/. The capture group is setup with .* before username which means capture everything up to Username. Once it sees username it will continue to capture everything up until the next comma. So if all of your events are formatted the same as your original it should capture what you are looking for. There are probably better ways to create the capture group since I'm am not a regex expert. I'd welcome anyone else's input on the regex.