How can I decrypt log events after forwarder sends them but before indexing

[dsharp1970]

To meet an internal security requirement I must encrypt data at rest in some locations. I'd like this data in Splunk but must obviously decrypt it first. I see three possibilities.

1) Decrypt before, or as, the universal forwarder sends the data to the indexer.

2) Interrupt the data flow and decrypt after the forwarder sends the data but before indexing.

3) Let the encrypted data be indexed and then decrypt at search time.

The first has an obvious issue in that it requires the decrypt key be on, or accessible from, the 'secure' location and mostly defeats having the data encrypted to begin with. It would seem the third option would create a lot of extra work on the search heads and there will be hundred of millions of these log entires that would greatly compound the issue.

The best option would seem to be the second but I don't see any way to interrupt the data flow. I know there are sed scripts that I can call using config in props.conf that doesn't seem flexible enough to solve this. Any one have a clever way of solving this problem?

[martin_mueller]

You might make option two work with a bit (a lot) of routing trickery.

Have the forwarders send the events with some encrypted payload to the indexers, using a sourcetype "foo-encrypted".
Set up routing for such sourcetypes to take an exit out of Splunk's index queue before the actual indexing, for example syslogout.
Send those events to a "decryption daemon" on your indexers that listens to the events routed off from the index queue and decrypts them.
Have the "decryption daemon" send the clear-text events back to Splunk, using a sourcetype "foo" that now gets sent along the regular indexing route.

Note, this a rough back-of-a-napkin draft... to actually implement this there surely is some more thinking and tinkering to be done.

[Ayn]

No, you cannot do this using option 2. Splunk has only very basic logic for transforming events based on regular expressions before indexing, nothing more.