topic Re: Is it possible to pseudonymize incoming data in Splunk? in Getting Data In

Is it possible to pseudonymize incoming data in Splunk?

schose — Tue, 23 Aug 2016 19:48:05 GMT

Hi forum,

I would like to know if and how it is possible to pseudonymise incoming data in Splunk. I know that I can anonymize data by applying a regex for an incoming sourcetype.

This procedure is removing information from the data. I would need something like applying a hash function to a certain type of data at parsing/index time.

Thanks for your help in advance,

Andreas

Re: Is it possible to pseudonymize incoming data in Splunk?

JDukeSplunk — Wed, 24 Aug 2016 16:10:02 GMT

This sounds like a job for SEDCMD in props.conf. I don't have an exact answer for you, but here are some breadcrumbs.

https://answers.splunk.com/answers/210096/how-to-configure-sedcmd-in-propsconf.html
https://answers.splunk.com/answers/323853/masking-ip-in-propsconf-using-sedcmd.html

Re: Is it possible to pseudonymize incoming data in Splunk?

Masa — Thu, 01 Sep 2016 01:00:08 GMT

No such a built-in feature in Splunk as of now. I recommend to file an enhancement request .

It is good to provide good use case when you file an enhancement request.

Re: Is it possible to pseudonymize incoming data in Splunk?

fbourel — Tue, 21 Aug 2018 08:52:16 GMT

Hi there,

I face the same issue/requirement. A good use case is nowadays when we use Splunk on sensitive incoming data that needs pseudonymisation, in order to be compliant with the European General Data Protection Regulation (GDPR).

Regards,

Re: Is it possible to pseudonymize incoming data in Splunk?

inventsekar — Tue, 21 Aug 2018 09:09:26 GMT

i think, the "Anonymize data" splunk document produces the exact output..
https://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Anonymizedata

For example, if you have a log file called accounts.log that contains Social Security and credit card numbers:

...ss=123456789, cc=1234-5678-9012-3456 ss=123456790, cc=2234-5678-9012-3457 ss=123456791, cc=3234-5678-9012-3458 ss=123456792, cc=4234-5678-9012-3459 ...
And you want to mask the fields, so that they appear like this:

...
ss=XXXXX6789, cc=XXXX-XXXX-XXXX-3456
ss=XXXXX6790, cc=XXXX-XXXX-XXXX-3457
ss=XXXXX6791, cc=XXXX-XXXX-XXXX-3458
ss=XXXXX6792, cc=XXXX-XXXX-XXXX-3459
...

Re: Is it possible to pseudonymize incoming data in Splunk?

fbourel — Tue, 21 Aug 2018 09:32:45 GMT

The need here is to pseudonymize and not anonymise which is different. Therefore the need is to be able to trace someone uniquely regardless of who he is namely. Anonymisation will lose traceability between events by replacing valuable information with "just" XXXX characters.

Regards,

Re: Is it possible to pseudonymize incoming data in Splunk?

inventsekar — Tue, 21 Aug 2018 10:00:52 GMT

how you can pseudonymize?!?! i mean, you want to pseudonymize only one string (only one ip address or SSN number, etc) or multiple strings?!?! i think you need to create "tokens" manually and using this token, do anonymize manually..

For other readers, this will help others to understand pseudonymization VS anonymization -
https://www.protegrity.com/pseudonymization-vs-anonymization-help-gdpr/

Re: Is it possible to pseudonymize incoming data in Splunk?

fbourel — Tue, 21 Aug 2018 12:00:49 GMT

Thanks for the link and its clarity.

Pseudonymisation in Splunk is not built-in, so one must rely on external programs to pseudonymise incoming raw data (one or several strings). I have found a Splunk app related to that issue: https://splunkbase.splunk.com/app/282/

I have also found a talk at the Splunk Conf 2017 clearly addressing the problem and the possible solutions :

Personally, I have the possibility to pseudonymize the input data before any Splunk indexation, so maybe I'll head that way for now.

Re: Is it possible to pseudonymize incoming data in Splunk?

jpass — Mon, 06 May 2019 04:54:24 GMT

Use INGEST_EVAL and cryptographic functions to create a hash at index time.

Re: Is it possible to pseudonymize incoming data in Splunk?

ankitsync — Fri, 07 Aug 2020 12:46:29 GMT

Is it still the same in 2020, has the capability been enabled in Splunk for pseudonymization ?

Re: Is it possible to pseudonymize incoming data in Splunk?

isoutamo — Fri, 07 Aug 2020 14:46:14 GMT

you should add the idea here https://ideas.splunk.com/ideas

r. Ismo

Re: Is it possible to pseudonymize incoming data in Splunk?

to4kawa — Sat, 08 Aug 2020 00:15:03 GMT

https://community.splunk.com/t5/Getting-Data-In/Anonymize-data-from-JSON-File/m-p/502197/highlight/true#M85571

INGEST_EVAL can work for it.