topic Re: Help with Masking data in Splunk Search

Help with Masking data

nmohammed — Wed, 15 Aug 2018 07:51:50 GMT

We have application writing logs as Windows Events . There are 3 fields that we wanted to mask ..

Accept-Language=en-US,en;q=0.9
Authorization=Auth xcvftYUIOLKN2luc3QiOiJiZTExMTQwODkzIiwiZWxsaV91aWQiOiJFbmNvbXBhc3NcXGJlMTExNDA4OTNcXDU5NTg0NjI4NTQiLCAic2Vzc2lvbiI6ImJlMTExNDA4OTNfMjhiZTI3NTYtZjY3MC00NGVhLTk4MzktMmM2NTRmMzkzZDc4IiwgInNpdGVfaWQiOiIzNjUxMzEzMzcxIiwgImluaXRfa2V5IjoiIn0=
Host=enc-ez9.xzapi.com
Referer=https://portal.juniorkiio.com/site-app/?id=1234567
User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
base-URI=/reverseproxy
Enc-Session=cte235467_28be2756-f670-44ea-9839-2c654f393d78
http-method=GET

We want to mask values of Authorization, Referer and Enc-Session. I have tried masking one to see, if it works but haven't seen any success.

Following is my

props.conf

[es_prd_api]
TRANSFORMS-anonymize = authorization-anonymizer

transforms.conf

[authorization-anonymizer]
REGEX = (?m)^(.*)Authorization=(.*)$
FORMAT = $1Authorization=########$2
DEST_KEY = Message

Appreciate help and guidance.

Thanks

Re: Help with Masking data

jkat54 — Wed, 15 Aug 2018 10:28:33 GMT

Try this instead

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = _raw

Or this

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = Message

You’ll have to reload your data, it will not hanged the existing data that has already been indexed.

Re: Help with Masking data

nmohammed — Wed, 15 Aug 2018 13:32:44 GMT

Thanks @jkat54

I tried both the options and pushed the bundle , but it still doesn't mask the Authorization field.

Re: Help with Masking data

cpetterborg — Wed, 15 Aug 2018 14:58:29 GMT

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

Re: Help with Masking data

sudosplunk — Wed, 15 Aug 2018 15:07:47 GMT

Give this a try,

[authorization-anonymizer]
REGEX = (?m)^(Authorization=).*$
FORMAT = $1########
DEST_KEY = Message

Re: Help with Masking data

sudosplunk — Wed, 15 Aug 2018 15:12:59 GMT

I was about to suggest using SEDCMD. But I wasn't sure how to properly interpret below information, per splunk docs.

Restrictions for using the sed script to anonymize data
If you use the SEDCMD method to anonymize the data, the following restrictions apply:

> The SEDCMD script applies only to the _raw field at index time. With the regular expression transform, you can apply changes to other fields.
> You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

Re: Help with Masking data

jkat54 — Wed, 15 Aug 2018 15:54:15 GMT

Is data flowing through / From a heavy forwarder?

Re: Help with Masking data

jkat54 — Wed, 15 Aug 2018 15:57:11 GMT

This is incorrect:

You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

They can be used in same props file but they apply from top down in ascii numerical order.

That is to say if this is my data

“Hello Good World”

SEDCMD-aaa = s/Good/Bad/
SEDCMD-bbb = s/Bad/Terrible/

Would change the data to this:

“Hello Terrible World”

Re: Help with Masking data

nmohammed — Wed, 15 Aug 2018 18:45:50 GMT

No. from Universal Forwarder.

Re: Help with Masking data

nmohammed — Wed, 15 Aug 2018 20:42:27 GMT

SEDCMD did the trick.

Thanks cpetterborg and all others who helped with the inputs.

Re: Help with Masking data

nmohammed — Wed, 15 Aug 2018 20:43:16 GMT

@nittala_surya

tried this before SEDCMD , it did not work. We could see the values coming in plain text.