Issues with Splunk "scrub" command not anomymizing data correctly

dshakespeare_sp — Tue, 29 Sep 2020 14:35:03 GMT

A customer was using Splunk "scrub" command to anonymize sensitive data (eg user name) at search time. While this worked well, they found names were not anonymized at all. They wrote a search to highlight these (index=test | table _time user | eval _user=user | scrub user | eval orig_user=_user | stats values(user) as users count by orig_user)

As we can see Splunk does a good job of anonymizing the user names Except for "Sarah Hardy" with is only partially anonymized and "Mike Smith" which is not anonymized at at all.

Why is this happening?

Re: Issues with Splunk "scrub" command not anomymizing data correctly

dshakespeare_sp — Mon, 26 Jun 2017 09:43:40 GMT

This is actually a documentation issue and in fact scrub is actually as intended (but not as documented). I will try to explain 🙂

The "scrub" and "splunk anonymize" (used to anonymize diags) commands share a common library

The scrub documentation states:

public-terms
Syntax: public-terms=
Description: Specify a filename that includes the public terms to be anonymized.

private-terms
Syntax: private-terms=
Description: Specify a filename that includes the private terms to be anonymized.

name-terms
Syntax: name-terms=
Description: Specify a filename that includes names to be anonymized.

dictionary
Syntax: dictionary=
Description: Specify a filename that includes a dictionary of terms to be anonymized.

timeconfig
Syntax: timeconfig=
Description: Specify a filename that includes time configurations to be anonymized.

namespace
Syntax: namespace=
Description: Specify an application that contains the alternative files to use for anonymizing, instead of using the built-in anonymizing files.

The anonymize command states

public-terms file containing a list of locally-used words to NOT anonymize
(default= $SPLUNK_HOME/etc/anonymizer/public-terms.txt)

private-terms file containing a list of words to anonymize
(default= $SPLUNK_HOME/etc/anonymizer/private-terms.txt)

name-terms file containing a list of common English personal
names that Splunk uses to anonymize names with
(default= $SPLUNK_HOME/etc/anonymizer/names.txt)

dictionary file containing a global list of commonly-used
words to NOT anonymize - unless they are in the
private-terms file
(default= $SPLUNK_HOME/etc/anonymizer/dictionary.txt)

timestamp-config file that determines how timestamps are parsed
(default= $SPLUNK_HOME/etc/anonymizer/
anonymizer-time.ini)

Note that dictionary and public-terms in the anonymize documentation are documented as having the OPPOSITE affect as those in scrub. The correct action is in the anonymize documentation, ie Dictionary.txt and public-terms.txt contain a list of words NOT to anonymize unless they are in private-terms.txt

Surnames like Smith and Hardy are included in dictionary.txt as "smith" is an noun and a verb and "hardy" is an adverb.
"Mike Smith" fails on two accounts as both "smith" and "mike" are included in dictionary.txt. Adding "Mike Smith" to private-terms.txt resolves the issue.

topic Re: Issues with Splunk "scrub" command not anomymizing data correctly in Getting Data In

Issues with Splunk "scrub" command not anomymizing data correctly

Re: Issues with Splunk "scrub" command not anomymizing data correctly

The scrub documentation states:

The anonymize command states