Getting Data In

How do you replace _raw values for multiple fields?

Contributor

I'm trying to mask multiple fields from the raw results. Only one of the fields ends up masked in the raw. It seems I need to either do one statement that gets them all or something else. I've experimented with using a pattern with pipes and also naming the EVAL-_raw differently like EVAL-_raw1 = and EVAL-raw2 = but have not found a winning combination. If I only try to mask one value I have no issue, so I believe it has to do with me trying doing the replace on more than one _raw string at once. I'm really hoping there is an answer other than deleting logs out. Any assistance is appreciated. These events are already indexed and I just want to mask the sensitive data at search time via props.conf on SH.

[wineventlog]

##DOB mask
EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>","<DateOfBirth>##masked##</DateOfBirth>")

##SSN mask
EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<SSN\>[^\<]+\<\/SSN\>","<SSN>##masked##</SSN>")

##LicenseNumber mask
EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<LicenseNumber\>[^\<]+\<\/LicenseNumber\>","<LicenseNumber>##masked##</LicenseNumber>")

##VIN mask
EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<VIN\>[^\<]+\<\/VIN\>","<VIN>##masked##</VIN>")
0 Karma
1 Solution

Contributor

The following props.conf worked in all modes (Verbose, Smart, Fast). It also redacts the data in all display modes such as List or Raw. I know the data will remain on disk and it would be better to do at index time. This is a good option for a quick mask with follow up conversations pending.

[wineventlog]


EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"<REDACTED>")


EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"<REDACTED>")


EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"<REDACTED>")


EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"<REDACTED>")

#Replace raw

EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>([^\<]+)","<\1><REDACTED>")
EVAL-Message = replace(Message,"(.+)","<REDACTED>")

View solution in original post

0 Karma

Contributor

The following props.conf worked in all modes (Verbose, Smart, Fast). It also redacts the data in all display modes such as List or Raw. I know the data will remain on disk and it would be better to do at index time. This is a good option for a quick mask with follow up conversations pending.

[wineventlog]


EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"<REDACTED>")


EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"<REDACTED>")


EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"<REDACTED>")


EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"<REDACTED>")

#Replace raw

EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>([^\<]+)","<\1><REDACTED>")
EVAL-Message = replace(Message,"(.+)","<REDACTED>")

View solution in original post

0 Karma

Esteemed Legend

There is no sense doing this at search time; do it at index-time like this:

[wineventlog]
SEDCMD-StripPII = s/<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>/<\1>##masked##<\\\3>/g

You can do it at search time similarly, like this (but I think that is silly, as it is trivially defeated):

[wineventlog]
EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>", "<\1>###<\\\3>")

Contributor

Thanks, this was to discover options after ingest other than pipe to delete or export, delete, re-ingest.

0 Karma

Revered Legend

Since all your eval trying to update same field (_raw), only last one would be effective. You can confirm that by running a btool command against that sourcetype.

Again, These search time mask will only apply if a user is running search on Smart/Verbose mode. If a user is running the search in fast mode, user can still see the original data. If you're OK with that fact, give this a try

[wineventlog]     
 ##DOB mask
 EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
 EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")

 ##SSN mask
 EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
 EVAL-SSN = if(isnull(SSN),NULL,"##masked##") 

 ##LicenseNumber mask
 EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
 EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##") 

 ##VIN mask
 EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
 EVAL-VIN = if(isnull(VIN),NULL,"##masked##")

 ##Raw data mask
 EVAL-_raw = replace(_raw,"(\<)(VIN|DateOfBirth|LicenseNumber|SSN)(\>)([^\<]+)", "\1\2\3##masked##")

Motivator

Could you explain why this is not working in fast mode?

0 Karma

Contributor

Thanks for help!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!