topic Re: Masking a data at search time in Getting Data In

Masking a data at search time

tmuthuk — Mon, 13 May 2013 18:49:30 GMT

There is log which we have n index has sensitive information like Date of birth /SSN /Credit card

I want to mask the sensitive information at execution time . Here is the sample log

...................
ns6:Date_Time2013-05-13T01:00:09/ns6:Date_Time
ns6:UpdateSourceLY/ns6:UpdateSource
ns7:Person
ns7:FirstNameTest/ns7:FirstName
ns7:LastNameuser/ns7:LastName
ns7:DateOfBirth1984-03-03/ns7:DateOfBirth
......................

I want to mask the data of birth as below

ns7:DateOfBirthXXXX-XX-XX/ns7:DateOfBirth

The transforms.conf i used is

REGEX = (?m)^(.):DateOfBirth>\d{4}-d{2}-d{2}(.)$
FORMAT = $1:DateOfBirth>####-##-##$2
DEST_KEY = _raw

But it is not working . Can anyone please help on this ?

Re: Masking a data at search time

MuS — Tue, 14 May 2013 09:44:20 GMT

Hi tmuthuk

your regex does not match the dateofbirth, d{4} looks for four d's and not four digits. I've played around at RegExr and found that this regex

\d{4}-\d{2}-\d{2}(?!T)

will match the dateofbirth in your provided log text.

cheers,
MuS

Re: Masking a data at search time

martin_mueller — Tue, 14 May 2013 10:59:17 GMT

The key issue here appears to be that you're defining a mask at index time, not at search time. See http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/Anonymizedatausingconfigurationfiles#Through_a_regex_transform for reference.

Re: Masking a data at search time

vkakani60 — Fri, 06 Oct 2017 16:53:35 GMT

what if the data is coming from dbconnect query ?

Re: Masking a data at search time

isoutamo — Tue, 29 Sep 2020 18:37:36 GMT

try to add a next definitions on search node's props.conf
--8<---
[your-source-type]
EXTRACT-DateOfBirth = \d{4}-\d{2}-\d{2}
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"####-##-##")
EVAL-_raw = replace(_raw,"\d{4}-\d{2}-\d{2}","####-##-##")
--8<---

And to metadata/default.meta
---8<--
[props/your-source-type/EXTRACT-DateOfBirth]
access = read : [ * ], write : [ admin ]
export = system
--8<--

These are working on our client's environment when we have to mask some other fields.

And if there are situation where you have to mask several fields on _raw then you should use next one on props.conf

--8<---
EVAL-_raw = replace(_raw, "(field1|field2|field3)=[([^]]+)]","\1=[####]")
EVAL-field1 = if(isnull(field1),NULL,"[####]")
EVAL-field2 = if(isnull(field2),NULL,"[####]")
EVAL-field3 = if(isnull(field3),NULL,"[####]")
--8<--

and of course to default.meta should be added all those definitions so no-one cannot update those by "accidentally".

where input is
field1=[asa],field2=[asdaa] etc.

regards Ismo

Re: Masking a data at search time

splunkreal — Fri, 17 Nov 2023 12:38:56 GMT

Hi, looks nice, thanks.