Splunk Search

Help with Masking data

Contributor

We have application writing logs as Windows Events . There are 3 fields that we wanted to mask ..

Accept-Language=en-US,en;q=0.9
Authorization=Auth xcvftYUIOLKN2luc3QiOiJiZTExMTQwODkzIiwiZWxsaV91aWQiOiJFbmNvbXBhc3NcXGJlMTExNDA4OTNcXDU5NTg0NjI4NTQiLCAic2Vzc2lvbiI6ImJlMTExNDA4OTNfMjhiZTI3NTYtZjY3MC00NGVhLTk4MzktMmM2NTRmMzkzZDc4IiwgInNpdGVfaWQiOiIzNjUxMzEzMzcxIiwgImluaXRfa2V5IjoiIn0=
Host=enc-ez9.xzapi.com
Referer=https://portal.juniorkiio.com/site-app/?id=1234567
User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
base-URI=/reverseproxy
Enc-Session=cte235467_28be2756-f670-44ea-9839-2c654f393d78
http-method=GET

We want to mask values of Authorization, Referer and Enc-Session. I have tried masking one to see, if it works but haven't seen any success.

Following is my

props.conf

[es_prd_api]
TRANSFORMS-anonymize = authorization-anonymizer

transforms.conf

[authorization-anonymizer]
REGEX = (?m)^(.*)Authorization=(.*)$
FORMAT = $1Authorization=########$2
DEST_KEY = Message

Appreciate help and guidance.

Thanks

0 Karma
1 Solution

SplunkTrust
SplunkTrust

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

View solution in original post

0 Karma

Motivator

Give this a try,

[authorization-anonymizer]
REGEX = (?m)^(Authorization=).*$
FORMAT = $1########
DEST_KEY = Message
0 Karma

Contributor

@nittala_surya

tried this before SEDCMD , it did not work. We could see the values coming in plain text.

0 Karma

SplunkTrust
SplunkTrust

You can do it all in props.conf with the following as an example:

[es_prd_api]
SEDCMD-anonauth = s/Authorization=.*/Authorization=########/
SEDCMD-anonrefer = s/Referer=.*/Referer=########/
SEDCMD-anonencs = s/Enc-Session=.*/Enc-Session=########/

View solution in original post

0 Karma

Contributor

SEDCMD did the trick.

Thanks cpetterborg and all others who helped with the inputs.

0 Karma

Motivator

I was about to suggest using SEDCMD. But I wasn't sure how to properly interpret below information, per splunk docs.

Restrictions for using the sed script to anonymize data
If you use the SEDCMD method to anonymize the data, the following restrictions apply:

> The SEDCMD script applies only to the _raw field at index time. With the regular expression transform, you can apply changes to other fields.
> You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.
0 Karma

SplunkTrust
SplunkTrust

This is incorrect:

You cannot use more than one SEDCMD type transformation for the same host, source, or source type in a single props.conf file.

They can be used in same props file but they apply from top down in ascii numerical order.

That is to say if this is my data

“Hello Good World”

SEDCMD-aaa = s/Good/Bad/
SEDCMD-bbb = s/Bad/Terrible/

Would change the data to this:

“Hello Terrible World”

SplunkTrust
SplunkTrust

Try this instead

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = _raw

Or this

 [authorization-anonymizer]
 REGEX = Authorization=(.*)$
 FORMAT = ########
 DEST_KEY = Message

You’ll have to reload your data, it will not hanged the existing data that has already been indexed.

0 Karma

Contributor

Thanks @jkat54

I tried both the options and pushed the bundle , but it still doesn't mask the Authorization field.

0 Karma

SplunkTrust
SplunkTrust

Is data flowing through / From a heavy forwarder?

0 Karma

Contributor

No. from Universal Forwarder.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!