I would like to know if and how it is possible to pseudonymise incoming data in Splunk. I know that I can anonymize data by applying a regex for an incoming sourcetype.
This procedure is removing information from the data. I would need something like applying a hash function to a certain type of data at parsing/index time.
Thanks for your help in advance,
This sounds like a job for SEDCMD in props.conf. I don't have an exact answer for you, but here are some breadcrumbs.
No such a built-in feature in Splunk as of now. I recommend to file an enhancement request .
It is good to provide good use case when you file an enhancement request.
I face the same issue/requirement. A good use case is nowadays when we use Splunk on sensitive incoming data that needs pseudonymisation, in order to be compliant with the European General Data Protection Regulation (GDPR).
i think, the "Anonymize data" splunk document produces the exact output..
For example, if you have a log file called accounts.log that contains Social Security and credit card numbers:
And you want to mask the fields, so that they appear like this:
... ss=XXXXX6789, cc=XXXX-XXXX-XXXX-3456 ss=XXXXX6790, cc=XXXX-XXXX-XXXX-3457 ss=XXXXX6791, cc=XXXX-XXXX-XXXX-3458 ss=XXXXX6792, cc=XXXX-XXXX-XXXX-3459 ...
The need here is to pseudonymize and not anonymise which is different. Therefore the need is to be able to trace someone uniquely regardless of who he is namely. Anonymisation will lose traceability between events by replacing valuable information with "just" XXXX characters.
how you can pseudonymize?!?! i mean, you want to pseudonymize only one string (only one ip address or SSN number, etc) or multiple strings?!?! i think you need to create "tokens" manually and using this token, do anonymize manually..
For other readers, this will help others to understand pseudonymization VS anonymization -
Thanks for the link and its clarity.
Pseudonymisation in Splunk is not built-in, so one must rely on external programs to pseudonymise incoming raw data (one or several strings). I have found a Splunk app related to that issue: https://splunkbase.splunk.com/app/282/
I have also found a talk at the Splunk Conf 2017 clearly addressing the problem and the possible solutions :
Personally, I have the possibility to pseudonymize the input data before any Splunk indexation, so maybe I'll head that way for now.