topic Re: Data masking in Getting Data In

Data masking

johnsasikumar — Tue, 19 Oct 2021 15:52:35 GMT

Hi,

Am trying to do an index time masking where my data is not in _raw but in a separate field A.

For example A field has the following data

"Path=/LoginUser Query=CrmId=ClientABC&
ContentItemId=TotalAccess&SessionId=3A1785URH117BEA&Ticket=646A1DA4STF896EE&
SessionTime=25368&ReturnUrl=http://www.clientabc.com, Method=GET,IP=209.51.249.195,
Content=", ""

I have applied transforms rules as below,

[session-anonymizer]
SOURCE_KEY = field:A
REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
FORMAT = $1SessionId=########$2
DEST_KEY = field:A

The problem is when we give the DEST_KEY as _raw it is masked properly, But I need the masked data back to field A. How do we get this masked to field:A

I have also tried adding

[accepted_keys]

is_valid = field:A

Re: Data masking

richgalloway — Tue, 19 Oct 2021 20:49:20 GMT

Have you tried masking using SEDCMD? It's simpler than using transforms. Put this in props.conf:

[mysourcetype] SEDCMD-maskSessionID = s/SessionId=[^&]+/SessionId=########/g

Re: Data masking

johnsasikumar — Wed, 20 Oct 2021 02:20:57 GMT

@richgalloway

Thanks for looking into this.
the problem is my data is not in _raw but in field A.
it comes as an additional field from hec as indexed field.

so SEDCMD has its limitation of being applied directly on _raw and not on indexed field.

thats why I had to use the SOURCE_KEY = field:A

Re: Data masking

johnsasikumar — Wed, 20 Oct 2021 06:27:17 GMT

INGEST_EVAL with replace solved the issue