topic Re: How do you replace _raw values for multiple fields? in Getting Data In

How do you replace _raw values for multiple fields?

jgbricker — Tue, 29 Sep 2020 23:55:21 GMT

I'm trying to mask multiple fields from the raw results. Only one of the fields ends up masked in the raw. It seems I need to either do one statement that gets them all or something else. I've experimented with using a pattern with pipes and also naming the EVAL-_raw differently like EVAL-_raw1 = and EVAL-raw2 = but have not found a winning combination. If I only try to mask one value I have no issue, so I believe it has to do with me trying doing the replace on more than one _raw string at once. I'm really hoping there is an answer other than deleting logs out. Any assistance is appreciated. These events are already indexed and I just want to mask the sensitive data at search time via props.conf on SH.

[wineventlog]

##DOB mask
EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>","<DateOfBirth>##masked##</DateOfBirth>")

##SSN mask
EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<SSN\>[^\<]+\<\/SSN\>","<SSN>##masked##</SSN>")

##LicenseNumber mask
EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<LicenseNumber\>[^\<]+\<\/LicenseNumber\>","<LicenseNumber>##masked##</LicenseNumber>")

##VIN mask
EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<VIN\>[^\<]+\<\/VIN\>","<VIN>##masked##</VIN>")

Re: How do you replace _raw values for multiple fields?

somesoni2 — Mon, 01 Apr 2019 21:11:29 GMT

Since all your eval trying to update same field (_raw), only last one would be effective. You can confirm that by running a btool command against that sourcetype.

Again, These search time mask will only apply if a user is running search on Smart/Verbose mode. If a user is running the search in fast mode, user can still see the original data. If you're OK with that fact, give this a try

[wineventlog]     
 ##DOB mask
 EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
 EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")

 ##SSN mask
 EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
 EVAL-SSN = if(isnull(SSN),NULL,"##masked##") 

 ##LicenseNumber mask
 EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
 EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##") 

 ##VIN mask
 EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
 EVAL-VIN = if(isnull(VIN),NULL,"##masked##")

 ##Raw data mask
 EVAL-_raw = replace(_raw,"(\<)(VIN|DateOfBirth|LicenseNumber|SSN)(\>)([^\<]+)", "\1\2\3##masked##")

Re: How do you replace _raw values for multiple fields?

woodcock — Tue, 02 Apr 2019 04:54:19 GMT

There is no sense doing this at search time; do it at index-time like this:

[wineventlog]
SEDCMD-StripPII = s/<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>/<\1>##masked##<\\\3>/g

You can do it at search time similarly, like this (but I think that is silly, as it is trivially defeated):

[wineventlog]
EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>", "<\1>###<\\\3>")

Re: How do you replace _raw values for multiple fields?

jgbricker — Tue, 02 Apr 2019 12:19:23 GMT

Thanks, this was to discover options after ingest other than pipe to delete or export, delete, re-ingest.

Re: How do you replace _raw values for multiple fields?

jgbricker — Wed, 03 Apr 2019 20:23:57 GMT

The following props.conf worked in all modes (Verbose, Smart, Fast). It also redacts the data in all display modes such as List or Raw. I know the data will remain on disk and it would be better to do at index time. This is a good option for a quick mask with follow up conversations pending.

[wineventlog]


EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"<REDACTED>")


EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"<REDACTED>")


EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"<REDACTED>")


EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"<REDACTED>")

#Replace raw

EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>([^\<]+)","<\1><REDACTED>")
EVAL-Message = replace(Message,"(.+)","<REDACTED>")

Re: How do you replace _raw values for multiple fields?

jgbricker — Wed, 03 Apr 2019 20:25:14 GMT

Thanks for help!

Re: How do you replace _raw values for multiple fields?

HeinzWaescher — Mon, 21 Oct 2019 08:03:32 GMT

Could you explain why this is not working in fast mode?