Getting Data In

How do you replace _raw values for multiple fields?

jgbricker
Contributor

I'm trying to mask multiple fields from the raw results. Only one of the fields ends up masked in the raw. It seems I need to either do one statement that gets them all or something else. I've experimented with using a pattern with pipes and also naming the EVAL-_raw differently like EVAL-_raw1 = and EVAL-raw2 = but have not found a winning combination. If I only try to mask one value I have no issue, so I believe it has to do with me trying doing the replace on more than one _raw string at once. I'm really hoping there is an answer other than deleting logs out. Any assistance is appreciated. These events are already indexed and I just want to mask the sensitive data at search time via props.conf on SH.

[wineventlog]

##DOB mask
EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>","<DateOfBirth>##masked##</DateOfBirth>")

##SSN mask
EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<SSN\>[^\<]+\<\/SSN\>","<SSN>##masked##</SSN>")

##LicenseNumber mask
EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<LicenseNumber\>[^\<]+\<\/LicenseNumber\>","<LicenseNumber>##masked##</LicenseNumber>")

##VIN mask
EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"##masked##")
EVAL-_raw = replace(_raw,"\<VIN\>[^\<]+\<\/VIN\>","<VIN>##masked##</VIN>")
0 Karma
1 Solution

jgbricker
Contributor

The following props.conf worked in all modes (Verbose, Smart, Fast). It also redacts the data in all display modes such as List or Raw. I know the data will remain on disk and it would be better to do at index time. This is a good option for a quick mask with follow up conversations pending.

[wineventlog]


EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"<REDACTED>")


EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"<REDACTED>")


EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"<REDACTED>")


EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"<REDACTED>")

#Replace raw

EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>([^\<]+)","<\1><REDACTED>")
EVAL-Message = replace(Message,"(.+)","<REDACTED>")

View solution in original post

0 Karma

jgbricker
Contributor

The following props.conf worked in all modes (Verbose, Smart, Fast). It also redacts the data in all display modes such as List or Raw. I know the data will remain on disk and it would be better to do at index time. This is a good option for a quick mask with follow up conversations pending.

[wineventlog]


EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
EVAL-DateOfBirth = if(isnull(DateOfBirth),NULL,"<REDACTED>")


EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
EVAL-SSN = if(isnull(SSN),NULL,"<REDACTED>")


EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"<REDACTED>")


EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
EVAL-VIN = if(isnull(VIN),NULL,"<REDACTED>")

#Replace raw

EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>([^\<]+)","<\1><REDACTED>")
EVAL-Message = replace(Message,"(.+)","<REDACTED>")
0 Karma

woodcock
Esteemed Legend

There is no sense doing this at search time; do it at index-time like this:

[wineventlog]
SEDCMD-StripPII = s/<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>/<\1>##masked##<\\\3>/g

You can do it at search time similarly, like this (but I think that is silly, as it is trivially defeated):

[wineventlog]
EVAL-_raw = replace(_raw,"<(DateOfBirth|SSN|LicenseNumber|VIN)>(.*?)<\/(DateOfBirth|SSN|LicenseNumber|VIN)>", "<\1>###<\\\3>")

jgbricker
Contributor

Thanks, this was to discover options after ingest other than pipe to delete or export, delete, re-ingest.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Since all your eval trying to update same field (_raw), only last one would be effective. You can confirm that by running a btool command against that sourcetype.

Again, These search time mask will only apply if a user is running search on Smart/Verbose mode. If a user is running the search in fast mode, user can still see the original data. If you're OK with that fact, give this a try

[wineventlog]     
 ##DOB mask
 EXTRACT-DOB = \<DateOfBirth\>(?<DateOfBirth>[^\<]+)\<\/DateOfBirth\>
 EVAL-DOB = if(isnull(DateOfBirth),NULL,"##masked##")

 ##SSN mask
 EXTRACT-SSN = \<SSN\>(?<SSN>[^\<]+)\<\/SSN\>
 EVAL-SSN = if(isnull(SSN),NULL,"##masked##") 

 ##LicenseNumber mask
 EXTRACT-LicenseNumber = \<LicenseNumber\>(?<LicenseNumber>[^\<]+)\<\/LicenseNumber\>
 EVAL-LicenseNumber = if(isnull(LicenseNumber),NULL,"##masked##") 

 ##VIN mask
 EXTRACT-VIN = \<VIN\>(?<VIN>[^\<]+)\<\/VIN\>
 EVAL-VIN = if(isnull(VIN),NULL,"##masked##")

 ##Raw data mask
 EVAL-_raw = replace(_raw,"(\<)(VIN|DateOfBirth|LicenseNumber|SSN)(\>)([^\<]+)", "\1\2\3##masked##")

HeinzWaescher
Motivator

Could you explain why this is not working in fast mode?

0 Karma

jgbricker
Contributor

Thanks for help!

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...