Regex help to mask data

[vishalduttauk]

I have to ingest some data so i've created a field called customer data and the regex works fine - ^[0-9]{16}.{249}(?<customer_information>.{174}).

As it contains PII data i need to mask it but keep the format of that event so the 174 characters within the customer_information field news to show as ####

Ive created this within the props.conf file but I can't get the data to be shown as ###. can you help?

[mask_customer_data]
DEST_KEY = _raw
REGEX = ^[0-9]{16}.{249}(?<customer_information>.{174})
FORMAT = $1CI##############################################################################################################################################################################

[gcusello]

Hi @vishalduttauk,

your regex is to extract a field not to replace a part of an event with a masquerading string, you can find more information about data masking at https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata

so please trysomething like this

props.conf:
[your_sourcetype]
TRANSFORMS-anonymizer = session-anonymizer


transforms.conf:
[session-anonymizer]
REGEX = ^[0-9]{16}.{249}(.{174})
FORMAT = ^[0-9]{16}.{249}(\*{174})
DEST_KEY = _raw

Ciao.

Giuseppe

[vishalduttauk]

Hi Giuseppe,

Thanks for your reply. I updated the props.conf file and created the transforms.conf file as per your reply. I have tried uploading some sample data (csv) and this is what happens:

[gcusello]

Hi @vishalduttauk,

please try this:

REGEX = ^([0-9]{16}.{249})(.{174})
FORMAT = ^$1(\*{174})

Giuseppe

[vishalduttauk]

Hi @gcusello ,

Thanks for your help so far but its not working yet. I have attached a sample record if that helps?

[gcusello]

Hi @vishalduttauk,

could you share a sample of your logs to mask (obviously renamed!).

ciao.

Giuseppe

[vishalduttauk]

Hi @gcusello,

I used the html editor to paste in an example and keep the format of the text:

0000440987654321    2901012001000000000002650380000767 2                                                                                                                                                       000000 000000 00000000                                   VMr Vishal S Dutta                      001 Street Road                 Town                                                                                            AA11AA  000000001011999   000000000 00000000                          0000000M

[gcusello]

Hi @vishalduttauk,

let me understand, you want as output:

0000440987654321    2901012001000000000002650380000767 2                                                                                                                                                       000000 000000 00000000                                   ***********************************************************************************************************************************************************************

is it correct?

In this case, please try this:

REGEX = ^((\d+\s+){6})(.*)
FORMAT = ^($1)(\*{174})

Ciao.

Giuseppe

Ciao.

Giuseppe

[vishalduttauk]

Hi @gcusello 

This is the format:

0000440987654321    2901012001000000000002650380000767 2                                                                                                                                                       000000 000000 00000000                                    M#############################################################################################################################################################################  000000001011999   000000000 00000000                          0000000M

[gcusello]

Hi @vishalduttauk,

please try this:

REGEX = ^((\d+\s+){6})(.*)(\s+\d+\s+\d+\s+\d+\s+\w+)
FORMAT = ^($1)(\#{174})($3)

Ciao.

Giuseppe

[vishalduttauk]

Hi @gcusello 

This is what I see now:

Could something within the the props file that is causing issues?

This is what i have set up

[CRA_Consumer_Txt_data]
CHARSET = UTF-8
DATETIME_CONFIG = CURRENT
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TRANSFORMS-anonymizer = session-anonymizer
SHOULD_LINEMERGE = false
category = Custom
disabled = false
pulldown_type = true
EXTRACT-close_date = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}(?<close_date>[0-9]{8})[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-current_balance = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}(?<current_balance>[0-9]{7}).{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-status = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}(?<status>[0-9]).{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-flag = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}(?<flag>[ A-Z]).{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-dob = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}(?<dob>[0-9]{8}).{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-start_date = ^[0-9]{16}.{4}[0-9]{2}(?<start_date>[0-9]{8})[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}[0-9]{7}[A-Z]
EXTRACT-payment_type = ^[0-9]{16}.{4}[0-9]{2}[0-9]{8}[0-9]{8}[0-9]{9}[0-9]{7}.{1}[0-9].{208}[ A-Z].{182}[0-9]{8}.{13}[0-9]{8}.{26}(?<payment_type>[0-9]{7}[A-Z])
EXTRACT-customer_information = ^[0-9]{16}.{249}(?<customer_information>.{174})

[vishalduttauk · Accepted solution]

Hi @gcusello,

I found a solution which is the regex below. Thanks for your help though ! 🙂

REGEX = (^[0-9]{16}.{249}).{174}(.+)

FORMAT = $1##############################################################################################################################################################################$2

DEST_KEY = _raw