Is it possible to pseudonymize incoming data in Splunk?

[schose]

Hi forum,

I would like to know if and how it is possible to pseudonymise incoming data in Splunk. I know that I can anonymize data by applying a regex for an incoming sourcetype.

This procedure is removing information from the data. I would need something like applying a hash function to a certain type of data at parsing/index time.

Thanks for your help in advance,

Andreas

[jpass]

Use INGEST_EVAL and cryptographic functions to create a hash at index time.

[inventsekar]

i think, the "Anonymize data" splunk document produces the exact output..
https://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Anonymizedata

For example, if you have a log file called accounts.log that contains Social Security and credit card numbers:

...
ss=123456789, cc=1234-5678-9012-3456
ss=123456790, cc=2234-5678-9012-3457
ss=123456791, cc=3234-5678-9012-3458
ss=123456792, cc=4234-5678-9012-3459
...
And you want to mask the fields, so that they appear like this:

...
ss=XXXXX6789, cc=XXXX-XXXX-XXXX-3456
ss=XXXXX6790, cc=XXXX-XXXX-XXXX-3457
ss=XXXXX6791, cc=XXXX-XXXX-XXXX-3458
ss=XXXXX6792, cc=XXXX-XXXX-XXXX-3459
... 

[fbourel]

The need here is to pseudonymize and not anonymise which is different. Therefore the need is to be able to trace someone uniquely regardless of who he is namely. Anonymisation will lose traceability between events by replacing valuable information with "just" XXXX characters.

Regards,

[inventsekar]

how you can pseudonymize?!?! i mean, you want to pseudonymize only one string (only one ip address or SSN number, etc) or multiple strings?!?! i think you need to create "tokens" manually and using this token, do anonymize manually..

For other readers, this will help others to understand pseudonymization VS anonymization -
https://www.protegrity.com/pseudonymization-vs-anonymization-help-gdpr/

[fbourel]

Thanks for the link and its clarity.

Pseudonymisation in Splunk is not built-in, so one must rely on external programs to pseudonymise incoming raw data (one or several strings). I have found a Splunk app related to that issue: https://splunkbase.splunk.com/app/282/

I have also found a talk at the Splunk Conf 2017 clearly addressing the problem and the possible solutions :

• link: http://conf.splunk.com/sessions/2017-sessions.html#search=obfuscation
• pdf: https://conf.splunk.com/files/2017/slides/data-obfuscation-and-field-protection-in-splunk.pdf

Personally, I have the possibility to pseudonymize the input data before any Splunk indexation, so maybe I'll head that way for now.

[ankitsync]

Is it still the same in 2020,  has the capability been enabled in Splunk for pseudonymization ?  

[to4kawa]

https://community.splunk.com/t5/Getting-Data-In/Anonymize-data-from-JSON-File/m-p/502197/highlight/t...

INGEST_EVAL can work for it.

[isoutamo]

Hi

you should add the idea here https://ideas.splunk.com/ideas

r. Ismo

[Masa]

No such a built-in feature in Splunk as of now. I recommend to file an enhancement request .

It is good to provide good use case when you file an enhancement request.

[fbourel]

Hi there,

I face the same issue/requirement. A good use case is nowadays when we use Splunk on sensitive incoming data that needs pseudonymisation, in order to be compliant with the European General Data Protection Regulation (GDPR).

Regards,

[JDukeSplunk]

This sounds like a job for SEDCMD in props.conf. I don't have an exact answer for you, but here are some breadcrumbs.

https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html
https://answers.splunk.com/answers/323853/masking-ip-in-propsconf-using-sedcmd.html