Annoymising IP but have a unique value for each IP

[PeterLai1]

Hi

We want to annoymise IPs, so far we can get it to replace all IP with x.x.x.x BUT we want to replace the IP with a unique value for each IP, so that we can see how many unique visitors and look up what they were doing without seeing any customer information.

Ideally, we want to do something like for (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$ | sha256sum
REGEX = (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$
FORMAT =$1 sha256sum the IP
DEST_KEY = _raw

[Damien_Dallimor]

How is the data getting into Splunk ? If it is being sent by TCP/UDP/HTTP ... then my PDI App's powerful preprocessor framework may be able to help , some examples here of pre processing , http://www.baboonbones.com/blog/get-binary-data-splunk/

[gcusello]

Hi PeterLai1,
we had a similar problem and, following a Splunk people hint, we pre-parsed logs with an external batch procedure in PHP, encrypting one field with an external certificate.
In this way we anonymized field value and at the same time we're able in every moment to uniquely identify each value of the encrypted field with the reverse procedure.

Bye.
Giuseppe

[PeterLai1]

Hey, thanks for reply,

I got confused, as in your constantly scanning the file and update the encrypt the IP? How did you ensure that splunk doesn't get the pre-encrypted file?

Or as in you encrypted the access log and ourput to access_encrypted.log and splunk then monitors the access_encrypted.log file?

[gcusello]

Hi PeterLai1,
we use the following process:

• we receive logs from syslog,
• we write them in a file (outside Splunk) named with date and time,
• we encrypt the requested field,
• we ingest parsed file in Splunk.

I understand that it isn't an easy process but this is the only way to solve our requirement.

Bye.
Giuseppe