Getting Data In

Is it possible to pseudonymize incoming data in Splunk?

schose
Builder

Hi forum,

I would like to know if and how it is possible to pseudonymise incoming data in Splunk. I know that I can anonymize data by applying a regex for an incoming sourcetype.

This procedure is removing information from the data. I would need something like applying a hash function to a certain type of data at parsing/index time.

Thanks for your help in advance,

Andreas

jpass
Contributor

Use INGEST_EVAL and cryptographic functions to create a hash at index time.

0 Karma

inventsekar
Ultra Champion

i think, the "Anonymize data" splunk document produces the exact output..
https://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Anonymizedata

For example, if you have a log file called accounts.log that contains Social Security and credit card numbers:

...
ss=123456789, cc=1234-5678-9012-3456
ss=123456790, cc=2234-5678-9012-3457
ss=123456791, cc=3234-5678-9012-3458
ss=123456792, cc=4234-5678-9012-3459
...

And you want to mask the fields, so that they appear like this:

...
ss=XXXXX6789, cc=XXXX-XXXX-XXXX-3456
ss=XXXXX6790, cc=XXXX-XXXX-XXXX-3457
ss=XXXXX6791, cc=XXXX-XXXX-XXXX-3458
ss=XXXXX6792, cc=XXXX-XXXX-XXXX-3459
... 

fbourel
Explorer

The need here is to pseudonymize and not anonymise which is different. Therefore the need is to be able to trace someone uniquely regardless of who he is namely. Anonymisation will lose traceability between events by replacing valuable information with "just" XXXX characters.

Regards,

0 Karma

inventsekar
Ultra Champion

how you can pseudonymize?!?! i mean, you want to pseudonymize only one string (only one ip address or SSN number, etc) or multiple strings?!?! i think you need to create "tokens" manually and using this token, do anonymize manually..

For other readers, this will help others to understand pseudonymization VS anonymization -
https://www.protegrity.com/pseudonymization-vs-anonymization-help-gdpr/

0 Karma

fbourel
Explorer

Thanks for the link and its clarity.

Pseudonymisation in Splunk is not built-in, so one must rely on external programs to pseudonymise incoming raw data (one or several strings). I have found a Splunk app related to that issue: https://splunkbase.splunk.com/app/282/

I have also found a talk at the Splunk Conf 2017 clearly addressing the problem and the possible solutions :

Personally, I have the possibility to pseudonymize the input data before any Splunk indexation, so maybe I'll head that way for now.

0 Karma

ankitsync
Explorer

Is it still the same in 2020,  has the capability been enabled in Splunk for pseudonymization ?  

0 Karma

to4kawa
Ultra Champion
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

you should add the idea here https://ideas.splunk.com/ideas

r. Ismo

0 Karma

Masa
Splunk Employee
Splunk Employee

No such a built-in feature in Splunk as of now. I recommend to file an enhancement request .

It is good to provide good use case when you file an enhancement request.

0 Karma

fbourel
Explorer

Hi there,

I face the same issue/requirement. A good use case is nowadays when we use Splunk on sensitive incoming data that needs pseudonymisation, in order to be compliant with the European General Data Protection Regulation (GDPR).

Regards,

0 Karma

JDukeSplunk
Builder

This sounds like a job for SEDCMD in props.conf. I don't have an exact answer for you, but here are some breadcrumbs.

https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html
https://answers.splunk.com/answers/323853/masking-ip-in-propsconf-using-sedcmd.html

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...