I've had quite a good look around the internet and have been unable to find an answer to this question. This question in particular touches on it, but the performance comparison is left unanswered. We are thinking about moving away from Splunk UF to an open source solution, which will likely only support HEC. Before making this change I'd like to know any consequences on performance/resource usage on the indexers.
What are the impacts on resource usage and index/search performance between UF and HEC?
If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.
You're mixing two different things.
One is HEC vs plain TCP for receiving data from sources. And here HEC is much better choice - its performance is great (whereas plain TCP inputs have its share of problems as I remember), you can create many different HEC endpoints on a single port and differentiate inputs by the token, you can dynamically overwrite metadata and add your own fields with HEC and so on.
Another thing is your own HEC-supplying "forwarder" vs. Splunk's UF. It's more tricky. Sure, you can get - for example - rsyslog instance to read files from disk and push events via HEC to your Splunk installation. It might even have some advantages over monitor input in UF (you can do much funky stuff in rsyslog that you can do in UF like filtering and advanced manipulation). But it's not a standardized approach, it will be harder to debug and you won't get support for such installation as easily as asking a question about UF here on Answers. And splunk's internal S2S protocol (the default mode of sending data from UF to indexers/HFs) is also very efficient.
So in general - HEC as input is a very efficient thing and can be used for relatively high throughputs. And you can put a load-balancer in front of your Splunk infrastructure if needed, you can offload TLS operations to an external solution - it's a standard HTTP after all. But should you replace UF... well, that would need more consideration.
HEC is very performant. Splunk Connect for Syslog (SC4S) uses it.
Thanks for the reply @richgalloway , but that doesn't answer my question at all. I am not interested in how performant HEC is in absolute terms (I'm not even sure "very" is a particularly meaningful measure of performance). Only when compared to UF to Splunk using TCP.
If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.