Getting Data In

Is HEC as performant as TCP?

tyates_ctm
Explorer

I've had quite a good look around the internet and have been unable to find an answer to this question. This question in particular touches on it, but the performance comparison is left unanswered. We are thinking about moving away from Splunk UF to an open source solution, which will likely only support HEC. Before making this change I'd like to know any consequences on performance/resource usage on the indexers.

What are the impacts on resource usage and index/search performance between UF and HEC?

Labels (2)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

You're mixing two different things.

One is HEC vs plain TCP for receiving data from sources. And here HEC is much better choice - its performance is great (whereas plain TCP inputs have its share of problems as I remember), you can create many different HEC endpoints on a single port and differentiate inputs by the token, you can dynamically overwrite metadata and add your own fields with HEC and so on.

Another thing is your own HEC-supplying "forwarder" vs. Splunk's UF. It's more tricky. Sure, you can get - for example - rsyslog instance to read files from disk and push events via HEC to your Splunk installation. It might even have some advantages over monitor input in UF (you can do much funky stuff in rsyslog that you can do in UF like filtering and advanced manipulation). But it's not a standardized approach, it will be harder to debug and you won't get support for such installation as easily as asking a question about UF here on Answers. And splunk's internal S2S protocol (the default mode of sending data from UF to indexers/HFs) is also very efficient.

So in general - HEC as input is a very efficient thing and can be used for relatively high throughputs. And you can put a load-balancer in front of your Splunk infrastructure if needed, you can offload TLS operations to an external solution - it's a standard HTTP after all. But should you replace UF... well, that would need more consideration.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

HEC is very performant.  Splunk Connect for Syslog (SC4S) uses it.

---
If this reply helps you, Karma would be appreciated.
0 Karma

tyates_ctm
Explorer

Thanks for the reply @richgalloway , but that doesn't answer my question at all. I am not interested in how performant HEC is in absolute terms (I'm not even sure "very" is a particularly meaningful measure of performance). Only when compared to UF to Splunk using TCP.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you're wondering if switching from S2S to HTTP (or vice versa) for UF->indexer communication would cause performance problem in network transmission layer, I'd say that you'll probably sooner hit performance limit with your indexers not able to keep up with parsing and writing events to disk.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...