Getting Data In

How to configure a heavy forwarder to anonymize data before sending logs to an indexer?

KevinAdu
Explorer

Still a bit new to Splunk but here goes my question.
My setup is pretty simple, it consists of a heavy forwarder sending logs to an indexer.

On the heavy forwarder, I want to mask some data before I send it to the index data. An example of the event looks like this:

Example Event

9/12/2014 17:21 Shawn Michaels 21.00 2013210345537512

I want to mask the last series of digits like this:

201321######7512

The edited config on my heavy forwarder is using the props.conf and transforms.conf. They look like this:

props.conf

[test-masking]
TRANSFORMS-masking = mask

transforms.conf

[mask]
REGEX = (.*)\s\d{16}$
FORMAT = $1\s\d{6}######\d{4}$2
DEST_KEY = _raw

The issue is that the data gets forwarded fine but the data doesn't seemed to have changed in anyway.

Does anything look strange?

0 Karma
1 Solution

KevinAdu
Explorer

I think I found out the problem. 2 things:

1) Yes my regex was definitely wrong for the REGEX and FORMAT fields.

2) Since my file wasn't getting updated there was nothing to forward and hence the changes wouldn't be applied.

View solution in original post

0 Karma

KevinAdu
Explorer

I think I found out the problem. 2 things:

1) Yes my regex was definitely wrong for the REGEX and FORMAT fields.

2) Since my file wasn't getting updated there was nothing to forward and hence the changes wouldn't be applied.

0 Karma

KevinAdu
Explorer

Oh I see, FORMAT doesn't use regex at all.
I just tried with the new FORMAT value and with SEDCMD but still no changes.

Could it possibly be that it's not reading the props or transforms files at all?
Is there anyway to check if it's doing so?

0 Karma

Ayn
Legend

A crude way of debugging it would be to setup a very simple transform just to make sure it's working, and then go from there. You can see which settings splunk "sees" by running the following command in Splunk's bin folder on the heavy forwarder:

splunk cmd btool props list --debug

And correspondingly for transforms.conf:

splunk cmd btool transforms list --debug

Also you might want to make sure that you're really running a heavy forwarder and not a light forwarder. Oh and if the data you're masking is coming from another heavy Splunk instance from the beginning this won't work.

0 Karma

Ayn
Legend

Also your output format looks weird. FORMAT is not denoted in regex. You probably want something like

REGEX = (.*\s\d{6})\d{6}(\d{4})$
FORMAT = $1######$2

Or for that matter, why not use SEDCMD:

props.conf:

[yoursourcetype]
SEDCMD-maskstuff = s/(\d{6})\d{6}(\d{4})$/\1#####$2/
0 Karma

Ayn
Legend

Is the sourcetype for your data really "test-masking"?

0 Karma

KevinAdu
Explorer

Yes it is. It says that both in the inputs.conf on the forwarder and in the event when searched for.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...