Help with SEDCMD in Props.conf

[dfurtaw]

Hi All,

I'm banging my head against a wall attempting to figure out why a SEDCMD inside of a props.conf on a UF isn't wanting to strip out the value I tell it to. We are wanting to strip out a hashed value from a log that is inside of a bracket (example below), as well as the brackets, with the SEDCMD. I am able to successfully test this command inside of the searchhead, but when I place it inside of the props.conf on the UF, I don't see it successfully implemented. I'm sure I'm missing something pretty simple. I've tried quite a few variations of this and no luck. Could anyone help me or possibly give me a hint as to what I could be doing wrong? Thank you all.

| rex mode=sed field=_raw "s/\[ecid: .+?\]//g"

[log4j]
SEDCMD-random=s/\[ecid: .+?\]//g

Sourcetype: log4j

[2020-06-24T10:02:08.590-04:00] [Server] [NOTIFICATION] [] [] [tid: 394025] [userId: <anonymous>] [ecid: 3956b675-4930-42d5-9e7d-94ca9013d2ea-0037ac42,0:26:74:38:2010:52:52:71:38] [APP: oraclediagent2] [partition-name: DOMAIN] [tenant-name: GLOBAL] [oracle.odi.runtime.MrepExtId: 38392028449]

[richgalloway]

Universal Forwarders don't support SEDCMD. Put that props.conf setting on your indexers.

[dfurtaw]

Thanks for the reply Rich! 

I recall in the past (6 or so months ago), I was able to place a SEDCMD in the props on a UF and saw the stripping of data. Did this change recently? By placing it in a props on the indexers, will this allow the data to be stripped BEFORE it enters the licensing phase? We are hoping to remove this large amount of unnecessary data before it hits this stage to limit ingestion.

[richgalloway]

Are you sure it was a UF you used in the past and not a heavy forwarder (HF)? HFs support SEDCMD.
Yes, using SEDCMD on the indexers strips data before it is counted against your license.

[dfurtaw]

Awesome. Thanks!

Yes, it was on the UF of our Syslog relay farm. It was a SEDCMD that obfuscated some sensitive data. Host -> Syslog -> Splunk Cloud

[richgalloway · Accepted solution]

Try this SEDCMD on your UF.

SEDCMD-ecid = s/(.*?)\[ecid: .+?\](.*)/\1\2/

[dfurtaw]

A little late on my reply, but it worked. Thanks Rich! I guess in some cases, we can SED on the UF.