Data Anonimization - Multiple transforms not working for single _raw event

nareshinsvu — Thu, 17 Sep 2020 07:37:01 GMT

Hi Punters,

I am facing issues with Data Anonimization. Below are my conf files. My transforms.conf anonimizes the data if my _raw event have any one regex pattern. But it's not anonimizing my _raw event if it has both the regex patterns. Need help please.

xml-anonymizer also doesn't work if my _raw event is having JSON message. But it works fine if the _raw event is a normal line.

props.conf

[dp_logs_multiline]
CHECK_METHOD = modtime
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}
category = Custom
disabled = false
pulldown_type = 1
MAX_TIMESTAMP_LOOKAHEAD = 24
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
TIME_PREFIX = ^
TRANSFORMS-anonymize = json-anonymizer, xml-anonymizer
ANNOTATE_PUNCT = false
TRUNCATE = 100000
MAX_EVENTS = 10000

transforms.conf

[json-anonymizer]
REGEX = (?ms)^(.*\"[sS]hippingAddress\"\s+\:\s+\{)[\s\S]*?(\}.*)$
FORMAT = $1#########JSON PCC DATA ANONIMIZED#############$2
REPEAT_MATCH = true
MV_ADD = true
DEST_KEY = _raw

[xml-anonymizer]
REGEX = (?ms)^(.*\<[bB]illTo\>)[\s\S]*?(\<\/[rR]equestMessage\>.*)$
FORMAT = $1#########XML PCC DATA ANONIMIZED#############$2
REPEAT_MATCH = true
MV_ADD = true
DEST_KEY = _raw

Re: Data Anonimization - Multiple transforms not working for single _raw event

thambisetty — Thu, 17 Sep 2020 12:26:07 GMT

Did you try two SEDCMD-class1, SEDCMD-class2. you don't need to have transforms.conf. Having multiple transforms will always be a problem.

topic Data Anonimization - Multiple transforms not working for single _raw event in Getting Data In

Data Anonimization - Multiple transforms not working for single _raw event

Re: Data Anonimization - Multiple transforms not working for single _raw event