topic Re: Encrypt data during anonymization in Getting Data In

Encrypt data during anonymization

hmozaffari — Tue, 19 May 2015 19:59:11 GMT

Referring to instruction of anonymization in page bellow:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedatausingconfigurationfiles

During indexing, instead of replacing a field value with literals, I would like to apply a function on it (for example encrypt it)

[session-anonymizer]
REGEX = (?m)^(.*)SessionId=\w+(\w{4}[&"].*)$
FORMAT = $1SessionId=########$2
DEST_KEY = _raw

For example instead of replacing SessionId=3A1785URH117BEA with SessionId=######## , I would like to replace it with a runtime value result of applying a function (like encryption function ).

This way I'll have a mechanism to get the original values if needed.

Has anybody come up with a solution for that.
Thanks

Re: Encrypt data during anonymization

woodcock — Tue, 19 May 2015 20:24:31 GMT

You are going to have to do this with a pre-parser (outside of Splunk); it is pretty easy.

Re: Encrypt data during anonymization

gesman — Mon, 28 Sep 2020 19:55:24 GMT

You mean: log.txt -> postprocessed to -> log_processed.txt -> indexing only log_processed.txt files ?

Re: Encrypt data during anonymization

woodcock — Wed, 20 May 2015 03:30:24 GMT

Yes, exactly.

Re: Encrypt data during anonymization

hmozaffari — Wed, 20 May 2015 12:28:23 GMT

Well, it works when batch processing files. But in case of real time monitoring files or TCP/UDP it is ideal to leave encryption to Splunk.

Re: Encrypt data during anonymization

mihenn — Tue, 03 Apr 2018 15:44:59 GMT

How do you do this in Splunk? I can't find any encryption function.

Re: Encrypt data during anonymization

Tapan_12345 — Tue, 23 Oct 2018 05:53:51 GMT

I am also searching for something similar. My requirement is I should be able to decrypt fields or rex pattern by supplying "KEY" on the search box . I did some search and found the best way to do is to write custom search command and feed the search result to this search command by eval function. The underlining decryption may be written in python sdk using mapper.

Please let me know

Re: Encrypt data during anonymization

jpass — Mon, 06 May 2019 04:40:26 GMT

Splunk 6.6 introduced cryptographic functions md5() sha1() etc.

You can encrypt fields at index time using these functions along with INGEST_EVAL.

You can also use calculated fields in props.conf to encrypt fields at search time. But one can easily view the source of the log data to see unencrypted values if done at search time.

See: https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/CryptographicFunctions

Re: Encrypt data during anonymization

mihenn — Mon, 06 May 2019 05:34:00 GMT

This way you can generate hashes. But it isn´t possible to revert this to get the original value. There is no password or certificate based encryption implemented in Splunk yet. As far as I know.

Re: Encrypt data during anonymization

woodcock — Sat, 11 May 2019 19:32:34 GMT

There is a hot new product called Cribl that is a swiss-army-knife to backfill all of the things that splunk should do but doesn't/can't. I passed this on to them and they should comment (@clintsharp, @dritan).

Re: Encrypt data during anonymization

manuelostertag — Fri, 13 Sep 2019 13:28:31 GMT

Hi Gregg,

thanks for the hint, it seems this tool could solve my problem described at 771002.

I will take a look at this tool.