Splunk Search

Help with Masking data

nmohammed
Contributor

We have application writing logs as Windows Events . There are 3 fields that we wanted to mask ..

Accept-Language=en-US,en;q=0.9
Authorization=Auth xcvftYUIOLKN2luc3QiOiJiZTExMTQwODkzIiwiZWxsaV91aWQiOiJFbmNvbXBhc3NcXGJlMTExNDA4OTNcXDU5NTg0NjI4NTQiLCAic2Vzc2lvbiI6ImJlMTExNDA4OTNfMjhiZTI3NTYtZjY3MC00NGVhLTk4MzktMmM2NTRmMzkzZDc4IiwgInNpdGVfaWQiOiIzNjUxMzEzMzcxIiwgImluaXRfa2V5IjoiIn0=
Host=enc-ez9.xzapi.com
Referer=https://portal.juniorkiio.com/site-app/?id=1234567
User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
base-URI=/reverseproxy
Enc-Session=cte235467_28be2756-f670-44ea-9839-2c654f393d78
http-method=GET

We want to mask values of Authorization, Referer and Enc-Session. I have tried masking one to see, if it works but haven't seen any success.

Following is my

props.conf

[es_prd_api]
TRANSFORMS-anonymize = authorization-anonymizer

transforms.conf

[authorization-anonymizer]
REGEX = (?m)^(.*)Authorization=(.*)$
FORMAT = $1Authorization=########$2
DEST_KEY = Message

Appreciate help and guidance.

Thanks

0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

View solution in original post

0 Karma

sudosplunk
Motivator

Give this a try,

[authorization-anonymizer]
REGEX = (?m)^(Authorization=).*$
FORMAT = $1########
DEST_KEY = Message
0 Karma

nmohammed
Contributor

@nittala_surya

tried this before SEDCMD , it did not work. We could see the values coming in plain text.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

View solution in original post

0 Karma

nmohammed
Contributor

SEDCMD did the trick.

Thanks cpetterborg and all others who helped with the inputs.

0 Karma

sudosplunk
Motivator

I was about to suggest using SEDCMD. But I wasn't sure how to properly interpret below information, per splunk docs.

Restrictions for using the sed script to anonymize data
If you use the SEDCMD method to anonymize the data, the following restrictions apply:

> The SEDCMD script applies only to the _raw field at index time. With the regular expression transform, you can apply changes to other fields.
> You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.
0 Karma

jkat54
SplunkTrust
SplunkTrust

This is incorrect:

You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

They can be used in same props file but they apply from top down in ascii numerical order.

That is to say if this is my data

“Hello Good World”

SEDCMD-aaa = s/Good/Bad/
SEDCMD-bbb = s/Bad/Terrible/

Would change the data to this:

“Hello Terrible World”

jkat54
SplunkTrust
SplunkTrust

Try this instead

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = _raw

Or this

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = Message

You’ll have to reload your data, it will not hanged the existing data that has already been indexed.

0 Karma

nmohammed
Contributor

Thanks @jkat54

I tried both the options and pushed the bundle , but it still doesn't mask the Authorization field.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Is data flowing through / From a heavy forwarder?

0 Karma

nmohammed
Contributor

No. from Universal Forwarder.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!