Getting Data In

Is it possible to pseudonymize incoming data in Splunk?

Contributor

Hi forum,

I would like to know if and how it is possible to pseudonymise incoming data in Splunk. I know that I can anonymize data by applying a regex for an incoming sourcetype.

This procedure is removing information from the data. I would need something like applying a hash function to a certain type of data at parsing/index time.

Thanks for your help in advance,

Andreas

Contributor

Use INGEST_EVAL and cryptographic functions to create a hash at index time.

0 Karma

Champion

i think, the "Anonymize data" splunk document produces the exact output..
https://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Anonymizedata

For example, if you have a log file called accounts.log that contains Social Security and credit card numbers:

...
ss=123456789, cc=1234-5678-9012-3456
ss=123456790, cc=2234-5678-9012-3457
ss=123456791, cc=3234-5678-9012-3458
ss=123456792, cc=4234-5678-9012-3459
...

And you want to mask the fields, so that they appear like this:

...
ss=XXXXX6789, cc=XXXX-XXXX-XXXX-3456
ss=XXXXX6790, cc=XXXX-XXXX-XXXX-3457
ss=XXXXX6791, cc=XXXX-XXXX-XXXX-3458
ss=XXXXX6792, cc=XXXX-XXXX-XXXX-3459
... 

Explorer

The need here is to pseudonymize and not anonymise which is different. Therefore the need is to be able to trace someone uniquely regardless of who he is namely. Anonymisation will lose traceability between events by replacing valuable information with "just" XXXX characters.

Regards,

0 Karma

Champion

how you can pseudonymize?!?! i mean, you want to pseudonymize only one string (only one ip address or SSN number, etc) or multiple strings?!?! i think you need to create "tokens" manually and using this token, do anonymize manually..

For other readers, this will help others to understand pseudonymization VS anonymization -
https://www.protegrity.com/pseudonymization-vs-anonymization-help-gdpr/

0 Karma

Explorer

Thanks for the link and its clarity.

Pseudonymisation in Splunk is not built-in, so one must rely on external programs to pseudonymise incoming raw data (one or several strings). I have found a Splunk app related to that issue: https://splunkbase.splunk.com/app/282/

I have also found a talk at the Splunk Conf 2017 clearly addressing the problem and the possible solutions :

Personally, I have the possibility to pseudonymize the input data before any Splunk indexation, so maybe I'll head that way for now.

0 Karma

Engager

Is it still the same in 2020,  has the capability been enabled in Splunk for pseudonymization ?  

0 Karma

Ultra Champion
0 Karma

Champion

Hi

you should add the idea here https://ideas.splunk.com/ideas

r. Ismo

0 Karma

Splunk Employee
Splunk Employee

No such a built-in feature in Splunk as of now. I recommend to file an enhancement request .

It is good to provide good use case when you file an enhancement request.

0 Karma

Explorer

Hi there,

I face the same issue/requirement. A good use case is nowadays when we use Splunk on sensitive incoming data that needs pseudonymisation, in order to be compliant with the European General Data Protection Regulation (GDPR).

Regards,

0 Karma

Builder

This sounds like a job for SEDCMD in props.conf. I don't have an exact answer for you, but here are some breadcrumbs.

https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html
https://answers.splunk.com/answers/323853/masking-ip-in-propsconf-using-sedcmd.html

0 Karma