How to use SEDCMD to anonymize a field after automatic lookup from a CSV file at index-time?

[joarsvensson]

I want to do an automatic lookup from a CSV file on index time, and add new fields to the event. I got this working, but what if I want to anonymize the field used as lookup key afterwards?

Using this won't work since it seem to happen prior to the lookup runs:

props.conf

[default]
SEDCMD-anonymize = s/username=(......)/username=XXXXXX/g

Help appreciated!

[koshyk]

Hope Splunk enabled a similar option for "tokenisation" of certain fields at index time (eg credit card numbers for apple pay)

[woodcock · Accepted solution]

It cannot be done without augmenting the data at Index-Time to include the lookup details. Lookups happen at Search-Time ALWAYS.

[joarsvensson]

Thank you for clarifying! So I need to populate the data prior to indexing, in order for this to work.

[woodcock]

Yes, think of it this way: any field created at Index-Time must be based off of a continuous string inside of the event itself (e.g. field X starts as position Y and ends at position Z) or in the meta-data for the event (e.g. source). This is how all Index-Time fields are defined and there is not (and probably never will be) any exception. Once I realized this, my thinking about fields became much more clear.

[joarsvensson]

Does no one have a solution or guidance to this? Help is very much appreciated!