topic Re: Transforms.conf - Hide values or make them anonymous in Splunk Search

Transforms.conf - Hide values or make them anonymous

celsohso — Wed, 30 Jul 2014 13:48:11 GMT

I have a log that look like this:

<ReceivedPermissions>EMULATION = [ EMULATEANOTHERUSER = Deny ], APPLICATION = [ PRIV FILE FDIC CAS = Deny, SESSION CLEAN UP = Deny, PRIV FILE IRS IBFD = Deny, WEB HOSTED CLIENTID IPAD = Grant,

I want to remove all Deny(Eg: ORG PRINTER SELECT = Deny)

On my transforms.conf I have

[removedeny]
REGEX = ^([A-Za-z0-9\S\s]+\s=\sDeny,)$
FORMAT = $1$2
DEST_KEY = _raw

On my props.conf I have

REPORT-removedeny= removedeny

But it is still not working: Do I need to use the field name, or change my regex? am I applying the proper user of Transform?

Thank you,

Re: Transforms.conf - Hide values or make them anonymous

somesoni2 — Wed, 30 Jul 2014 13:57:47 GMT

See these links for masking your data (based on regex)

http://answers.splunk.com/answers/62374/anonymize-the-sensitive-data-no-gaurantee-in-splunk

http://answers.splunk.com/answers/76825/anonymize-data-using-regex-transform

Re: Transforms.conf - Hide values or make them anonymous

somesoni2 — Wed, 30 Jul 2014 17:32:55 GMT

If we take this as sample log entry, what should be the expected output??
Input:
EMULATION = [ EMULATEANOTHERUSER = Deny ], APPLICATION = [ PRIV FILE FDIC CAS = Deny, SESSION CLEAN UP = Deny, PRIV FILE IRS IBFD = Deny, WEB HOSTED CLIENTID IPAD = Grant,

Output??
EMULATION WEB HOSTED CLIENTID IPAD = Grant,

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Wed, 30 Jul 2014 17:39:44 GMT

Yes, the granted are the only ones I want to see as result,

Re: Transforms.conf - Hide values or make them anonymous

somesoni2 — Wed, 30 Jul 2014 18:21:07 GMT

Give this a try. No transforms.conf change needed.

props.conf

[YourSourceType]
..
Other configurations
..
SEDCMD-deny = s/(\[)*(\w+\s+)+=\sDeny(,|\s)//g

I tried with following sample data and below that is the outpt I received.
Sample data:

<ReceivedPermissions>EMULATION = [ EMULATEANOTHERUSER = Deny, SESSION CLEAN UP = Deny ], APPLICATION = [ PRIV FILE FDIC CAS = Deny, PRIV FILE IRS IBFD = Deny, WEB HOSTED CLIENTID IPAD = Grant,
<ReceivedPermissions>EMULATION = [ EMULATEANOTHERUSER = Deny ], APPLICATION = [ PRIV FILE FDIC CAS = Deny, SESSION CLEAN UP = Deny, PRIV FILE IRS IBFD = Deny, WEB HOSTED CLIENTID IPAD = Grant,
<ReceivedPermissions>TEST = [ EMULATEANOTHERUSER = Deny ], APPLICATION = [ PRIV FILE FDIC CAS = Deny, SESSION CLEAN UP = Deny, PRIV FILE IRS IBFD = Deny, WEB HOSTED CLIENTID IPAD = Grant ]

Output after SEDCMD:

<ReceivedPermissions>TEST = [ ], APPLICATION = [ WEB HOSTED CLIENTID IPAD = Grant ]
<ReceivedPermissions>EMULATION = [ ], APPLICATION = [ WEB HOSTED CLIENTID IPAD = Grant,
<ReceivedPermissions>EMULATION = [ ], APPLICATION = [ WEB HOSTED CLIENTID IPAD = Grant,

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Wed, 30 Jul 2014 18:41:49 GMT

I was trying this same property when I got your message,
it seems to be a much easier solution. I am testing your regex at this moment, it seems to be working much better then mine was, and yours is actually a lot simpler too.
As soon as I finish my test I will let you know the results,

Thanks a lot, you have been really helpful!

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Wed, 30 Jul 2014 18:47:23 GMT

It worked great!

one thing though, I notice that your results did not have the big spaces that mine have. I think I might be able to fix that tweaking your regex. That is great man!

EMULATION = [ ], APPLICATION = [ WEB HOSTED CLIENTID IPAD = Grant, PRACINSIGHTPHONE = Grant, DESKTOP PRACLAW CORP = Grant, KCALERT MONTHLY = Grant, COINV ALERTS = Grant, ANNOTATIONS = Grant, DESKTOP PRACLAW EMP = Grant, DESKTOP PRACLAW CAP = Grant, MYBI- BLC ZONE = Grant, KEYCITE ALERTS = Grant, EMAIL DELIVERY = Grant, TAX KPMG USER = Grant,

Re: Transforms.conf - Hide values or make them anonymous

somesoni2 — Wed, 30 Jul 2014 19:54:13 GMT

Glad it helped. Let me know if there are any followup questions, else just mark the question answered.

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Fri, 01 Aug 2014 16:21:30 GMT

Two Question: Do you happen to know if:
-As far as performance goes, is there any difference in change the Transforms.conf or, add only SEDCMD on props.conf ?
-Also, the white spaces are related to Splunk way to deal with the Sed, or the regex need to be tweaked?

“EMULATION = [ ], APPLICATION = [ WEB HOSTED CLIENTID IPAD = Grant, PRACINSIGHTPHONE = Grant, DESKTOP PRACLAW CORP = Grant, KCALERT MONTHLY = Grant, COINV ALERTS = Grant, ANNOTATIONS = Grant, DESKTOP PRACLAW EMP = Grant, DESKTOP PRACLAW CAP = Grant, MYBI- BLC ZONE = Grant,

”
Thank you,

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Fri, 01 Aug 2014 16:49:31 GMT

Also, can this be test from the Splunk search page, so I can play with regex without being restarting splunk indexers
SEDCMD-deny = s/([)*(\w+\s+)+=\sDeny(,|\s)//g

Re: Transforms.conf - Hide values or make them anonymous

celsohso — Fri, 01 Aug 2014 17:05:35 GMT

we found the answer, we add and extra \s here sDeny(,\s|\s)
before
([)(\w+\s+)+=\sDeny(,|\s)
after
([)(\w+\s+)+=\sDeny(,\s|\s)
Thank you ,

Re: Transforms.conf - Hide values or make them anonymous

somesoni2 — Fri, 01 Aug 2014 18:13:15 GMT

For you first question, see this. Since you're doing multiple remove, SEDCMD is your guy.
http://answers.splunk.com/answers/9456/performance-difference-between-using-sedcmd-and-older-regextransforms-method.

Great job resolving the extra spaces issue. I was getting that too but somehow didn't show when pasted the result here.