Getting Data In

Is HEC as performant as TCP?

tyates_ctm
Explorer

I've had quite a good look around the internet and have been unable to find an answer to this question. This question in particular touches on it, but the performance comparison is left unanswered. We are thinking about moving away from Splunk UF to an open source solution, which will likely only support HEC. Before making this change I'd like to know any consequences on performance/resource usage on the indexers.

What are the impacts on resource usage and index/search performance between UF and HEC?

Labels (2)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

You're mixing two different things.

One is HEC vs plain TCP for receiving data from sources. And here HEC is much better choice - its performance is great (whereas plain TCP inputs have its share of problems as I remember), you can create many different HEC endpoints on a single port and differentiate inputs by the token, you can dynamically overwrite metadata and add your own fields with HEC and so on.

Another thing is your own HEC-supplying "forwarder" vs. Splunk's UF. It's more tricky. Sure, you can get - for example - rsyslog instance to read files from disk and push events via HEC to your Splunk installation. It might even have some advantages over monitor input in UF (you can do much funky stuff in rsyslog that you can do in UF like filtering and advanced manipulation). But it's not a standardized approach, it will be harder to debug and you won't get support for such installation as easily as asking a question about UF here on Answers. And splunk's internal S2S protocol (the default mode of sending data from UF to indexers/HFs) is also very efficient.

So in general - HEC as input is a very efficient thing and can be used for relatively high throughputs. And you can put a load-balancer in front of your Splunk infrastructure if needed, you can offload TLS operations to an external solution - it's a standard HTTP after all. But should you replace UF... well, that would need more consideration.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

HEC is very performant.  Splunk Connect for Syslog (SC4S) uses it.

---
If this reply helps you, Karma would be appreciated.
0 Karma

tyates_ctm
Explorer

Thanks for the reply @richgalloway , but that doesn't answer my question at all. I am not interested in how performant HEC is in absolute terms (I'm not even sure "very" is a particularly meaningful measure of performance). Only when compared to UF to Splunk using TCP.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.

Get Updates on the Splunk Community!

Monitoring MariaDB and MySQL

In a previous post, we explored monitoring PostgreSQL and general best practices around which metrics to ...

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...