Splunk Dev

Annoymising IP but have a unique value for each IP

PeterLai1
New Member

Hi

We want to annoymise IPs, so far we can get it to replace all IP with x.x.x.x BUT we want to replace the IP with a unique value for each IP, so that we can see how many unique visitors and look up what they were doing without seeing any customer information.

Ideally, we want to do something like for (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$ | sha256sum
REGEX = (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$
FORMAT =$1 sha256sum the IP
DEST_KEY = _raw

Tags (1)
0 Karma

Damien_Dallimor
Ultra Champion

How is the data getting into Splunk ? If it is being sent by TCP/UDP/HTTP ... then my PDI App's powerful preprocessor framework may be able to help , some examples here of pre processing , http://www.baboonbones.com/blog/get-binary-data-splunk/

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi PeterLai1,
we had a similar problem and, following a Splunk people hint, we pre-parsed logs with an external batch procedure in PHP, encrypting one field with an external certificate.
In this way we anonymized field value and at the same time we're able in every moment to uniquely identify each value of the encrypted field with the reverse procedure.

Bye.
Giuseppe

0 Karma

PeterLai1
New Member

Hey, thanks for reply,

I got confused, as in your constantly scanning the file and update the encrypt the IP? How did you ensure that splunk doesn't get the pre-encrypted file?

Or as in you encrypted the access log and ourput to access_encrypted.log and splunk then monitors the access_encrypted.log file?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi PeterLai1,
we use the following process:

  • we receive logs from syslog,
  • we write them in a file (outside Splunk) named with date and time,
  • we encrypt the requested field,
  • we ingest parsed file in Splunk.

I understand that it isn't an easy process but this is the only way to solve our requirement.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...