topic How do you anonymize two recognized fields in Splunk? in Splunk Search

How do you anonymize two recognized fields in Splunk?

dmacgillivray — Wed, 12 Nov 2014 17:47:38 GMT

Hello Splunkers,

I am trying to follow the logic from the below URL to anonymize some field data on the fly.
http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Anonymizedatausingconfigurationfiles

The following Splunk Search query gets all the fields for my output without regex.

sourcetype=ourdatasource userPassword | table index host source sourcetype authority user password userPassword

The next step for me is to try and anonymize this data.
Both regular expressions below are verified to work with just the rex fields below.

As I write this question I am starting to wonder if I even need a REGEX statement below
for these fields as they are all recognized fields in Splunk. Can someone confirm that maybe
I do not need a REGEX statement. Maybe something else?

Also I am not sure about the format statement if that is the case.

[ourdatasource]
TRANSFORMS-anonymize = userpass-anonymizer, pass-anonymizer

[userpass-anonymizer]
REGEX = (?i)^(?:[^;]*;){4}(?P[^=]+)
FORMAT = $1UserPassword=###$2
DEST_KEY = _raw

[pass-anonymizer]
REGEX = (?i)&(?P[^=]+)
FORMAT = $1password=########$2
DEST_KEY = _raw

Thanks,
Daniel

Re: How do you anonymize two recognized fields in Splunk?

aweitzman — Wed, 12 Nov 2014 18:16:39 GMT

The URL that you pointed to refers to anonymizing the data prior to it being indexed - if you do this, the actual data will never be viewable in Splunk, just the anonymized version. This may or may not be what you want.

If it is OK that only the anonymized version of the data is in Splunk, then you must anonymize it prior to it being indexed, which means you do, in fact, need the regexes. The reason is that indexing happens before Splunk tries to break up your data into discoverable fields. Once indexing happens, the data cannot be changed, so it would be searchable in Splunk if it had not been anonymized prior.

Re: How do you anonymize two recognized fields in Splunk?

dmacgillivray — Wed, 12 Nov 2014 18:42:06 GMT

Thanks for your comments on this subject. Unfortunately I have tried this, as I have completely removed the data on my test server which is pointing to an app location on /opt/splunk/etc/apps.

Then I tried again, reloaded the data and re-queried the data with the same results.

I have also moved this entire app over to /opt/splunk/etc/system/local just to see if it was a placement issue.
That also did not work out. If you can think of something else, let me know. Very interesting point you make about
the regex part of my question.

Re: How do you anonymize two recognized fields in Splunk?

aweitzman — Wed, 12 Nov 2014 19:33:26 GMT

All I can think of is that there might be an issue with your regex or FORMAT string. The primary thrust of my answer had to do with whether the regexes were necessary or not.

In terms of the actual regexes and FORMAT strings, sample raw data will be necessary to help you there.