How to configure a heavy forwarder to anonymize data before sending logs to an indexer?

[KevinAdu]

Still a bit new to Splunk but here goes my question.
My setup is pretty simple, it consists of a heavy forwarder sending logs to an indexer.

On the heavy forwarder, I want to mask some data before I send it to the index data. An example of the event looks like this:

Example Event

9/12/2014 17:21 Shawn Michaels 21.00 2013210345537512

I want to mask the last series of digits like this:

201321######7512

The edited config on my heavy forwarder is using the props.conf and transforms.conf. They look like this:

props.conf

[test-masking]
TRANSFORMS-masking = mask

transforms.conf

[mask]
REGEX = (.*)\s\d{16}$
FORMAT = $1\s\d{6}######\d{4}$2
DEST_KEY = _raw

The issue is that the data gets forwarded fine but the data doesn't seemed to have changed in anyway.

Does anything look strange?

[KevinAdu · Accepted solution]

I think I found out the problem. 2 things:

1) Yes my regex was definitely wrong for the REGEX and FORMAT fields.

2) Since my file wasn't getting updated there was nothing to forward and hence the changes wouldn't be applied.

[KevinAdu]

Oh I see, FORMAT doesn't use regex at all.
I just tried with the new FORMAT value and with SEDCMD but still no changes.

Could it possibly be that it's not reading the props or transforms files at all?
Is there anyway to check if it's doing so?

[Ayn]

A crude way of debugging it would be to setup a very simple transform just to make sure it's working, and then go from there. You can see which settings splunk "sees" by running the following command in Splunk's bin folder on the heavy forwarder:

splunk cmd btool props list --debug

And correspondingly for transforms.conf:

splunk cmd btool transforms list --debug

Also you might want to make sure that you're really running a heavy forwarder and not a light forwarder. Oh and if the data you're masking is coming from another heavy Splunk instance from the beginning this won't work.

[Ayn]

Also your output format looks weird. FORMAT is not denoted in regex. You probably want something like

REGEX = (.*\s\d{6})\d{6}(\d{4})$
FORMAT = $1######$2

Or for that matter, why not use SEDCMD:

props.conf:

[yoursourcetype]
SEDCMD-maskstuff = s/(\d{6})\d{6}(\d{4})$/\1#####$2/

[Ayn]

Is the sourcetype for your data really "test-masking"?

[KevinAdu]

Yes it is. It says that both in the inputs.conf on the forwarder and in the event when searched for.