topic Re: In the following string of data, how do you mask what follows the characters "cpf"? in Getting Data In

In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Fri, 30 Nov 2018 19:01:43 GMT

I hava a log on a Windows server like this: D:\SplunkTest\confidencial.log and on this log, I have data like this:

nome:user1,cpf:32861123800
nome:user1,cpf:32861123822

and I would like to mask all data after "cpf=" with XXXXX, so I configure /opt/splunk/etc/system/local/props.conf with script below:

[source::D:\\SplunkTest\\confidencial.log]

TRANSFORMS-anonymize = cpf-anonymize
root@LXPNHHSPLUNK01:/opt/splunk/etc/system/local#

and configure /opt/splunk/etc/system/local/transforms.conf with:

[cpf-anonymize]
REGEX = (?m)^(.*)cpf=\w+(\w{4}[&"].*)$
DEST_KEY = _raw
FORMAT = $1cpf=###########$2

However, after doing this configuration, I write in the log and nothing is masked. Could someone help me and point out where I'm going wrong?

Best Regards

Re: In the following string of data, how do you mask what follows the characters "cpf"?

bjoernjensen — Fri, 30 Nov 2018 19:49:18 GMT

Hi,

your regex does not look like it is written for you raw data? Moreover you might want to put that into an extra app and not into system/local.

Maybe something like this might go into your intended direction:

props.conf:
[source::D:\\SplunkTest\\confidencial.log]
TRANSFORMS-anonymize = cpf-anonymizer

transforms.conf:
[cpf-anonymizer]
REGEX = ^(.+?),cpf:.*$
FORMAT = $1,cpf:###########
DEST_KEY = _raw

Edit: simplified regex, added props.conf and removed single quotes as mentioned by @ddrillic, thanks.

This will take everything after ,cpf: until the end of the event and replace it by ###########.

Hope that helps.

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Mon, 03 Dec 2018 14:01:20 GMT

Hi @bjoernjensen,

Thanks for your help but I think that I still make something wrong. Look my files:

/opt/splunk/etc/system/local# cat props.conf
[source:D:\SplunkTest\confidencial.log]
TRANSFORMS-anonymize = cpf-anonymizer
/opt/splunk/etc/system/local# cat transforms.conf
[cpf-anonymizer]
REGEX = '^(.+)(,cpf:)(.+)$'
FORMAT = $1$2###########
DEST_KEY = _raw
/opt/splunk/etc/system/local#

Could you help me?

Re: In the following string of data, how do you mask what follows the characters "cpf"?

ddrillic — Mon, 03 Dec 2018 14:33:55 GMT

I believe it should read REGEX = ^(.+)(,cpf:)(.+)$, meaning, no single quotes...

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Mon, 03 Dec 2018 16:12:09 GMT

Not yet. Do I need to restart the service for the change to start working?
See the files:

/opt/splunk/etc/system/local# cat transforms.conf
[cpf-anonymizer]
REGEX = ^(.+)(,cpf:)(.+)$
FORMAT = $1$2###########
DEST_KEY = _raw

/opt/splunk/etc/system/local# cat props.conf
[source:D:\SplunkTest\confidencial.log]
TRANSFORMS-anonymize = cpf-anonymizer
/opt/splunk/etc/system/local#

Re: In the following string of data, how do you mask what follows the characters "cpf"?

bjoernjensen — Mon, 03 Dec 2018 16:26:07 GMT

Yes, and if you want to reindex, you have to set crcSalt ...
https://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf

Re: In the following string of data, how do you mask what follows the characters "cpf"?

bjoernjensen — Mon, 03 Dec 2018 17:41:19 GMT

In your props.conf stanza you have to type double colon (::) after source:
[source::<SOME_SOURCE>]

Re: In the following string of data, how do you mask what follows the characters "cpf"?

ddrillic — Mon, 03 Dec 2018 17:49:12 GMT

Great catch @bjoernjensen ; -)

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Mon, 03 Dec 2018 18:40:41 GMT

I followed your instructions but it still did not work even after the service restart.
See conf files:

/opt/splunk/etc/system/local# cat props.conf
[sourcetype::confidencial-too_small]
TRANSFORMS-anonymize = cpf-anonymizer

/opt/splunk/etc/system/local# cat transforms.conf
[cpf-anonymizer]
REGEX = ^(.+)(,cpf:)(.+)$
FORMAT = $1$2###########
DEST_KEY = _raw
/opt/splunk/etc/system/local#

I changed source by sourcetype to see if there is anything related to sorce, however both one and the other do not mask the data

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Mon, 03 Dec 2018 18:46:36 GMT

Where can I be wrong? What else could you do to solve it?

Re: In the following string of data, how do you mask what follows the characters "cpf"?

woodcock — Mon, 03 Dec 2018 20:40:53 GMT

You are taking the long way using the old approach; keep it simple like this in props.conf:

[confidencial-too_small]
SEDCMD-cpf_anonymizer = s/,cpf:\d+/,cpf:###########/

Deploy to the first full instance of splunk that handles the data (usually the Indexers but you might have HF) and restart all splunk instances there.

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Fri, 07 Dec 2018 18:40:23 GMT

I followed his suggestion however after having changed the props.conf restart the splunk but still did not work I write in the log file something like: "testing, cpf = 156766164343" and nothing is changed.

See the file: /opt/splunk/etc/system/local# cat props.conf

[sourcetype::confidencial-too_small]
SEDCMD-cpf_anonymizer = s/,cpf:\d+/,cpf:###########/

Re: In the following string of data, how do you mask what follows the characters "cpf"?

woodcock — Sun, 09 Dec 2018 00:32:44 GMT

Your stanza header is wrong; remove the sourcetype:: as indicated in my updated answer.

Re: In the following string of data, how do you mask what follows the characters "cpf"?

woodcock — Mon, 10 Dec 2018 06:46:20 GMT

Did this work for you, @justodaniel? Be sure to UpVote any helpful answers and click Accept if one worked.

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Thu, 27 Dec 2018 18:13:40 GMT

Sorry I had not seen the answer. I'll do what you said and I'll tell you the result here.

Re: In the following string of data, how do you mask what follows the characters "cpf"?

justodaniel — Thu, 27 Dec 2018 18:41:53 GMT

Thank you! worked perfectly.