topic Re: Regex help to mask data in Getting Data In

Regex help to mask data

vishalduttauk — Wed, 21 Sep 2022 11:30:57 GMT

I have to ingest some data so i've created a field called customer data and the regex works fine - ^[0-9]{16}.{249}(?<customer_information>.{174}).

As it contains PII data i need to mask it but keep the format of that event so the 174 characters within the customer_information field news to show as ####

Ive created this within the props.conf file but I can't get the data to be shown as ###. can you help?

[mask_customer_data]
DEST_KEY = _raw
REGEX = ^[0-9]{16}.{249}(?<customer_information>.{174})
FORMAT = $1CI##############################################################################################################################################################################

Re: Regex help to mask data

gcusello — Wed, 21 Sep 2022 12:00:28 GMT

Hi @vishalduttauk,

your regex is to extract a field not to replace a part of an event with a masquerading string, you can find more information about data masking at https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata

so please trysomething like this

props.conf: [your_sourcetype] TRANSFORMS-anonymizer = session-anonymizer transforms.conf: [session-anonymizer] REGEX = ^[0-9]{16}.{249}(.{174}) FORMAT = ^[0-9]{16}.{249}(\*{174}) DEST_KEY = _raw

Ciao.

Giuseppe

Re: Regex help to mask data

vishalduttauk — Wed, 21 Sep 2022 13:39:00 GMT

Hi Giuseppe,

Thanks for your reply. I updated the props.conf file and created the transforms.conf file as per your reply. I have tried uploading some sample data (csv) and this is what happens:

Re: Regex help to mask data

gcusello — Wed, 21 Sep 2022 14:53:55 GMT

Hi @vishalduttauk,

please try this:

REGEX = ^([0-9]{16}.{249})(.{174}) FORMAT = ^$1(\*{174})

Giuseppe

Re: Regex help to mask data

vishalduttauk — Thu, 22 Sep 2022 07:34:20 GMT

Hi @gcusello ,

Thanks for your help so far but its not working yet. I have attached a sample record if that helps?

Re: Regex help to mask data

gcusello — Thu, 22 Sep 2022 08:11:09 GMT

Hi @vishalduttauk,

could you share a sample of your logs to mask (obviously renamed!).

ciao.

Giuseppe

Re: Regex help to mask data

vishalduttauk — Thu, 22 Sep 2022 10:47:08 GMT

Hi @gcusello,

I used the html editor to paste in an example and keep the format of the text:

0000440987654321 2901012001000000000002650380000767 2 000000 000000 00000000 VMr Vishal S Dutta 001 Street Road Town AA11AA 000000001011999 000000000 00000000 0000000M

Re: Regex help to mask data

gcusello — Thu, 22 Sep 2022 11:02:32 GMT

Hi @vishalduttauk,

let me understand, you want as output:

0000440987654321 2901012001000000000002650380000767 2 000000 000000 00000000 ***********************************************************************************************************************************************************************

is it correct?

In this case, please try this:

REGEX = ^((\d+\s+){6})(.*) FORMAT = ^($1)(\*{174})

Ciao.

Giuseppe

Ciao.

Giuseppe

Re: Regex help to mask data

vishalduttauk — Thu, 22 Sep 2022 11:23:04 GMT

Hi @gcusello

This is the format:

0000440987654321 2901012001000000000002650380000767 2 000000 000000 00000000 M############################################################################################################################################################################# 000000001011999 000000000 00000000 0000000M

Re: Regex help to mask data

gcusello — Thu, 22 Sep 2022 12:09:00 GMT

Hi @vishalduttauk,

please try this:

REGEX = ^((\d+\s+){6})(.*)(\s+\d+\s+\d+\s+\d+\s+\w+) FORMAT = ^($1)(\#{174})($3)

Ciao.

Giuseppe

Re: Regex help to mask data

vishalduttauk — Fri, 23 Sep 2022 09:16:55 GMT

Hi @gcusello

This is what I see now:

Could something within the the props file that is causing issues?

This is what i have set up

[CRA_Consumer_Txt_data]
CHARSET = UTF-8
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TRANSFORMS-anonymizer = session-anonymizer
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true
EXTRACT-close_date = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}(?<close_date>[0-9]{8})[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-current_balance = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}(?<current_balance>[0-9]{7}).{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-status = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}(?<status>[0-9]).{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-flag = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}(?<flag>[ A-Z]).{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-dob = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}(?<dob>[0-9]{8}).{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-start_date = ^[0-9]{16}.{4}[0-9]{2}(?<start_date>[0-9]{8})[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-payment_type = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}(?<payment_type>[0-9]{7}[A-Z])
EXTRACT-customer_information = ^[0-9]{16}.{249}(?<customer_information>.{174})

Re: Regex help to mask data

vishalduttauk — Wed, 28 Sep 2022 12:47:32 GMT

Hi @gcusello,

I found a solution which is the regex below. Thanks for your help though ! 🙂

REGEX = (^[0-9]{16}.{249}).{174}(.+)

FORMAT = $1##############################################################################################################################################################################$2

DEST_KEY = _raw