Developing for Splunk Platform

Annoymising IP but have a unique value for each IP

PeterLai1
New Member

Hi

We want to annoymise IPs, so far we can get it to replace all IP with x.x.x.x BUT we want to replace the IP with a unique value for each IP, so that we can see how many unique visitors and look up what they were doing without seeing any customer information.

Ideally, we want to do something like for (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$ | sha256sum
REGEX = (?m)^(.)clientip=\d+.\d+.\d+.\d+ (.)$
FORMAT =$1 sha256sum the IP
DEST_KEY = _raw

Tags (1)
0 Karma

Damien_Dallimor
Ultra Champion

How is the data getting into Splunk ? If it is being sent by TCP/UDP/HTTP ... then my PDI App's powerful preprocessor framework may be able to help , some examples here of pre processing , http://www.baboonbones.com/blog/get-binary-data-splunk/

0 Karma

gcusello
Legend

Hi PeterLai1,
we had a similar problem and, following a Splunk people hint, we pre-parsed logs with an external batch procedure in PHP, encrypting one field with an external certificate.
In this way we anonymized field value and at the same time we're able in every moment to uniquely identify each value of the encrypted field with the reverse procedure.

Bye.
Giuseppe

0 Karma

PeterLai1
New Member

Hey, thanks for reply,

I got confused, as in your constantly scanning the file and update the encrypt the IP? How did you ensure that splunk doesn't get the pre-encrypted file?

Or as in you encrypted the access log and ourput to access_encrypted.log and splunk then monitors the access_encrypted.log file?

0 Karma

gcusello
Legend

Hi PeterLai1,
we use the following process:

  • we receive logs from syslog,
  • we write them in a file (outside Splunk) named with date and time,
  • we encrypt the requested field,
  • we ingest parsed file in Splunk.

I understand that it isn't an easy process but this is the only way to solve our requirement.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...