We are wondering if the Splunk forwarders care if SSL termination is not done at the indexer? We would like the forwarder to have the SSL cert of our NG firewall, which will then decrypt the traffic and send it to an indexer. Will this work? Does it need to be a direct connect from the fwd to the indexer?
I've never tried this, but I expect trouble. I think but am not 100% sure that Splunk uses SSL compression when sending data via SSL. So the forwarder would be sending compressed data, and then your firewall would have to both decrypt and decompress and then forward the decompressed stream in the clear to the indexer. This would make me concerned about your firewall's memory and CPU usage and so forth.
This is a configuration that will probably not be supported by the Splunk support team. Using a 3rd party load balancer is frowned upon from a support perspective because of how it affects the load balancing. I think this would fall into a very similar situation - you might be able to make it work but I'm not sure you would be in a supported configuration.
I would strongly suggest avoiding this approach and just let Splunk forward direct to the indexers without having any in-line data manipulation.