Help with Masking data

[nmohammed]

We have application writing logs as Windows Events . There are 3 fields that we wanted to mask ..

Accept-Language=en-US,en;q=0.9
Authorization=Auth xcvftYUIOLKN2luc3QiOiJiZTExMTQwODkzIiwiZWxsaV91aWQiOiJFbmNvbXBhc3NcXGJlMTExNDA4OTNcXDU5NTg0NjI4NTQiLCAic2Vzc2lvbiI6ImJlMTExNDA4OTNfMjhiZTI3NTYtZjY3MC00NGVhLTk4MzktMmM2NTRmMzkzZDc4IiwgInNpdGVfaWQiOiIzNjUxMzEzMzcxIiwgImluaXRfa2V5IjoiIn0=
Host=enc-ez9.xzapi.com
Referer=https://portal.juniorkiio.com/site-app/?id=1234567
User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
base-URI=/reverseproxy
Enc-Session=cte235467_28be2756-f670-44ea-9839-2c654f393d78
http-method=GET

We want to mask values of Authorization, Referer and Enc-Session. I have tried masking one to see, if it works but haven't seen any success.

Following is my

props.conf

[es_prd_api]
TRANSFORMS-anonymize = authorization-anonymizer

transforms.conf

[authorization-anonymizer]
REGEX = (?m)^(.*)Authorization=(.*)$
FORMAT = $1Authorization=########$2
DEST_KEY = Message

Appreciate help and guidance.

Thanks

[sudosplunk]

Give this a try,

[authorization-anonymizer]
REGEX = (?m)^(Authorization=).*$
FORMAT = $1########
DEST_KEY = Message

[nmohammed]

@nittala_surya

tried this before SEDCMD , it did not work. We could see the values coming in plain text.

[cpetterborg · Accepted solution]

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

[nmohammed]

SEDCMD did the trick.

Thanks cpetterborg and all others who helped with the inputs.

[sudosplunk]

I was about to suggest using SEDCMD. But I wasn't sure how to properly interpret below information, per splunk docs.

Restrictions for using the sed script to anonymize data
If you use the SEDCMD method to anonymize the data, the following restrictions apply:

> The SEDCMD script applies only to the _raw field at index time. With the regular expression transform, you can apply changes to other fields.
> You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

[jkat54]

This is incorrect:

You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

They can be used in same props file but they apply from top down in ascii numerical order.

That is to say if this is my data

“Hello Good World”

SEDCMD-aaa = s/Good/Bad/
SEDCMD-bbb = s/Bad/Terrible/

Would change the data to this:

“Hello Terrible World”

[jkat54]

Try this instead

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = _raw

Or this

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = Message

You’ll have to reload your data, it will not hanged the existing data that has already been indexed.

[nmohammed]

Thanks @jkat54

I tried both the options and pushed the bundle , but it still doesn't mask the Authorization field.

[jkat54]

Is data flowing through / From a heavy forwarder?

[nmohammed]

No. from Universal Forwarder.