topic Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders? in Getting Data In

After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Mon, 24 Apr 2017 14:18:13 GMT

Hi All

I have followed the regular expression method to anonymize data during indexing as mentioned in the below Splunk documentation.
https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Data/Anonymizedata

Path : {Splunk_home}/etc/system/local

props.conf entry:
[access_log]
TRANSFORMS-anonymize = cardType1-anonymizer, cardType2-anonymizer

transforms.conf entry:
[cardType1-anonymizer]
REGEX = (.*?)(37)\d{2}(-|%20)\d{6}(-|%20)\d{1}(.*)(37)\d{2}(-|%20)\d{6}(-|%20)\d{1}(.*?)$
FORMAT = $1$2##$3######$4#$5$6##$7######$8#$9
DEST_KEY = _raw

[cardType2-anonymizer]
REGEX = (.*?)(37)\d{2}(-|%20)\d{6}(-|%20)\d{1}(.*?)$
FORMAT = $1$2##$3######$4#$5
DEST_KEY = _raw

When I am loading data from Search Head UI using Settings > Add Data > Upload from My Computer the masking is working and card numbers are getting masked properly.
However when the same data is coming from universal forwarders installed on application servers the masking is not working.
In both cases I have the same sourcetype.
I am not able to understand what is it that I am missing.
Can anyone help me to resolve this.

Thanks
Nirmalya

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

gcusello — Mon, 24 Apr 2017 14:34:01 GMT

Hi nirmalya2006,
where do you inserted the props.conf to hide sensitive numbers?
you have to insert it in all the indexers.
Bye.
Giuseppe

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

adonio — Mon, 24 Apr 2017 14:35:04 GMT

please verify the inputs on forwarder gives the same sourcetype to match the satnza in props.conf
sourcetype = access_logs

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Mon, 24 Apr 2017 15:10:00 GMT

sourcetype is verified. As mentioned, data loaded locally is masked but not from the forwarder for same sourcetype

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Mon, 24 Apr 2017 15:13:25 GMT

I have only one indexer for the current testing that I am doing.
I have placed it in {splunk_home}/etc/system/local on the indexer as mentioned on the documentation.
Do you think I am missing something else.

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

gcusello — Mon, 24 Apr 2017 15:25:27 GMT

Hi Hi nirmalya2006,,
verify that sourcetypes in your inputs.conf are the same of your props.conf.

after verify your regexes.

after insert transforms command in two different rows:

TRANSFORMS-anonymize1 = cardType1-anonymizer
TRANSFORMS-anonymize2 = cardType2-anonymizer

Bye.
Giuseppe

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Tue, 25 Apr 2017 08:55:17 GMT

Verified sourcetypes and regex.
Also the transforms as you mentioned.
But still data uploaded from local is getting masked but the data that is being forwarded from the forwarders are not getting masked.

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

gcusello — Tue, 29 Sep 2020 13:49:37 GMT

hi nirmalya2006,
have you INDEXED_EXTRACTIONS data? (see http://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Data/Extractfieldsfromfileswithstructureddata#Forward_data_extracted_from_structured_data_files)
in this case you have to insert props.conf and trasforms.conf also in UFs (https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Data/Anonymizedata).

Bye.
Giuseppe

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Tue, 25 Apr 2017 11:26:26 GMT

Sorry Cusello.
Tried all that stuff and I am following the same documentation.
So just now I tried to remove all the regex and use only SEDCMD in props.conf
It included just replace anything that comes in with a random string.
Didn't work.
So it seems the props.conf is not even read, when the data comes in from forwarder.
Did the same thing on the universal forwarder also. But there also it seems to skip reading the props.conf file.
I am not using INDEXED_EXTRACTIONS as this is unstructured data and there is no delimiter that I can use for extractions.
I am at a loss for all options to mask the account numbers in the logs

Re: After editing props.conf, why is sensitive information not masked when data is coming from universal forwarders?

nirmalya2006 — Wed, 26 Apr 2017 10:55:01 GMT

Finally I got this working.
I had to contact my splunk infrastructure team and found that I have been making the changes in secondary indexer servers and primary search head servers.
As a result data ingested through search head was getting masked and data ingested through forwarders were not getting masked since the data from forwarders were hitting the primary indexer where the changes were not placed.

So, I had to make the changes in the primary indexers and the primary search head to get it working.