ignore specific parts of a log entry with props.conf and transforms.conf?

[jpena323]

Hi Splunkers!

Thank you so much for your help in advance!

I'm trying to ignore specific fields within a syslog entry from being forwarded to Splunk. This can not be done on the host level so I was looking towards the props/transforms for help. Unfortunately, I can not figure this one out.

I have a single line log entry from a Juniper SRX:

Dec 31 07:55:25 srx1 1 2014-12-31T07:55:25.244 srx1a RT_FLOW - RT_FLOW_SESSION_CREATE [junos@2644.1.1.1.2.35 source-address="10.170.72.52" source-port="51888" destination-address="10.5.80.36" destination-port="10050" service-name="None" nat-source-address="10.170.72.52" nat-source-port="51885" nat-destination-address="10.5.80.36" nat-destination-port="10050" src-nat-rule-name="None" dst-nat-rule-name="None" protocol-id="6" policy-name="TRUST-VPN-PERMIT" source-zone-name="TRUST" destination-zone-name="VPN" session-id-32="120129546" username="N/A" roles="N/A" packet-incoming-interface="reth1.104" application="UNKNOWN" nested-application="UNKNOWN" encrypted="UNKNOWN"]

That's a giant entry (I have lots of them!) and I'd like to eliminate specific fields that we have aren't interested in (i.e. "src-nat-rule-name="None") but keeping the rest of the log entry. Is this possible? I think specifying any field with regex will match (and null out) the entire line and that's not what I want. My files for reference:

props.conf

[source::/syslog/srx1/user.info.log]
TRANSFORMS-set= setnull

transforms.conf

[setnull]
REGEX = (?:source-zone.? |session-id.? |src-nat-rule-name.? |junos@.? |destination-zone.? |username.? |roles.? |packet-incoming.? |application.? |nested-application.? |encrypted.*? |RT_FLOW - )
DEST_KEY = queue
FORMAT = nullQueue

[somesoni2]

You can try SEDCMD command to remove unwanted fields/strings. add more similar SEDCMD lines for different fields.

E.g.

props.conf

[YourSourceType]
...Other parts...
...........
SEDCMD-removeunwanted1 = s/\s(src-nat-rule-name=\"[^\"]+\")//

[aweitzman]

I've done something similar to this. In transforms.conf you basically need to rewrite _raw with matching groups from your regex. So you'll write your regex so that some matching groups match what you want to keep, and others match what you want to discard. Then you append the matching groups you want to keep and apply them to _raw.

props.conf

[stanza]
TRANSFORM-keepthegoodstuff = keepthegoodstuff

transforms.conf

[keepthegoodstuff]
REGEX=^(.*) fieldtoremove1=\"(.*)\"(.*) fieldtoremove2=\"(.*)\"(.*)$
FORMAT=$1$3$5
DEST_KEY=_raw

Perhaps not exactly this, but you get the general idea. Hope this is helpful.