Masking a data at search time

[tmuthuk]

There is log which we have n index has sensitive information like Date of birth /SSN /Credit card

I want to mask the sensitive information at execution time . Here is the sample log

...................
ns6:Date_Time2013-05-13T01:00:09/ns6:Date_Time
ns6:UpdateSourceLY/ns6:UpdateSource
ns7:Person
ns7:FirstNameTest/ns7:FirstName
ns7:LastNameuser/ns7:LastName
ns7:DateOfBirth1984-03-03/ns7:DateOfBirth
......................

I want to mask the data of birth as below

ns7:DateOfBirthXXXX-XX-XX/ns7:DateOfBirth

The transforms.conf i used is

REGEX = (?m)^(.):DateOfBirth>\d{4}-d{2}-d{2}(.)$
FORMAT = $1:DateOfBirth>####-##-##$2
DEST_KEY = _raw

But it is not working . Can anyone please help on this ?

[isoutamo]

Hi

try to add a next definitions on search node's props.conf
--8<---
[your-source-type]
EXTRACT-DateOfBirth = \d{4}-\d{2}-\d{2}
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"####-##-##")
EVAL-_raw = replace(_raw,"\d{4}-\d{2}-\d{2}","####-##-##")
--8<---

And to metadata/default.meta
---8<--
[props/your-source-type/EXTRACT-DateOfBirth]
access = read : [ * ], write : [ admin ]
export = system
--8<--

These are working on our client's environment when we have to mask some other fields.

And if there are situation where you have to mask several fields on _raw then you should use next one on props.conf

--8<---
EVAL-_raw = replace(_raw, "(field1|field2|field3)=[([^]]+)]","\1=[####]")
EVAL-field1 = if(isnull(field1),NULL,"[####]")
EVAL-field2 = if(isnull(field2),NULL,"[####]")
EVAL-field3 = if(isnull(field3),NULL,"[####]")
--8<--

and of course to default.meta should be added all those definitions so no-one cannot update those by "accidentally".

where input is
field1=[asa],field2=[asdaa] etc.

regards Ismo

[splunkreal]

Hi, looks nice, thanks.

[martin_mueller]

The key issue here appears to be that you're defining a mask at index time, not at search time. See http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/Anonymizedatausingconfigurationfiles#Through_... for reference.

[vkakani60]

what if the data is coming from dbconnect query ?

[MuS]

Hi tmuthuk

your regex does not match the dateofbirth, d{4} looks for four d's and not four digits. I've played around at RegExr and found that this regex

\d{4}-\d{2}-\d{2}(?!T)

will match the dateofbirth in your provided log text.

cheers,
MuS