topic Re: How can we avoid the scaling issues with on-prem HEC subclusters? in Deployment Architecture

How can we avoid the scaling issues with on-prem HEC subclusters?

danielbb — Fri, 13 Dec 2024 18:49:02 GMT

On Splunk cloud, we can receive HEC ingestion directly to the cloud whereas on-prem we install distinct subclusters for HEC and struggle to scale them up with multiple down-times cases. I wonder whether it's possible for an on-prem installation to send HEC data directly to the indexers.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

PickleRick — Fri, 13 Dec 2024 21:06:55 GMT

I'm not sure what you mean by "subcluster". You can have HEC inputs directly on indexers (and have an LB in front of them) or can have a farm of HFs with HECs sending to the cluster.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

danielbb — Fri, 13 Dec 2024 21:20:35 GMT

I love of the idea of the - HEC inputs directly on indexers (and have an LB in front of them) !

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

isoutamo — Sat, 14 Dec 2024 00:14:12 GMT

You should look this https://docs.splunk.com/Documentation/SVA/current/Architectures/TopologyGuidance
It contains preferred Splunk architecture layouts.

You should remember that if you have lot of HEC inputs and you need to update/add those regularly this impacts your indexers it you are using those instead of HFs as HEC inputs.
For that reason I personally prefer to use couple of HFs with LB as a HEC cluster instead of configure those directly into indexers.

Here is some instructions how to tune HEC.
- https://community.splunk.com/t5/Getting-Data-In/What-are-the-best-HEC-perf-tuning-configs/m-p/601629
- https://community.splunk.com/t5/Getting-Data-In/Can-we-have-fewer-Heavy-Forwarders-than-Indexers/m-p/551485

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

PickleRick — Sat, 14 Dec 2024 09:55:50 GMT

+1 on that. While an input/parsing HF layer isn't exactly SVA I like this approach because it isolates inputs and index-time parsing settings from the indexers. So the only maintenance you need to do on the indexers is strictly about indexes and storage.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

dural_yyz — Sat, 14 Dec 2024 21:55:54 GMT

In a past environment I ran isolated HF's specifically for HEC with no other purpose. I was able to tune them up on HEC processing cause there was no conflicting use cases for the compute power. I had 2 sites with 2 HF's per site all acting in full HA behind site LB and local LB configurations. The HF's were pointed to the Deployment Servers so the HEC inputs and config could be updated in a central location with auto distribution.

To size up I would only have to stand up another HF in either location one at a time or in bulk. Have them point to the DS and confirm the local logs were found at the indexers. Then I could add the new server addresses to the server class for the HEC inputs/config to upload. Then update the LB to include the new server(s) in the pool for that particular site.

Very convenient and not difficult once setup - although not for novice users though. Plugging an HF to a DS can come with issues if not 100% aware of how apps are named.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

danielbb — Mon, 16 Dec 2024 15:47:12 GMT

Thank you all, since I have already the HEC cluster and it is with HFs, wouldn't it be better to have UFs since my understanding is that data flows quicker through UFs? and we don't process the data at all at this HEC cluster level.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

PickleRick — Mon, 16 Dec 2024 18:59:48 GMT

Officially HEC isn't supported on UF. I read several times that it works but never actually got to testing it myself.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

dural_yyz — Mon, 16 Dec 2024 20:38:31 GMT

I agree about recommending to avoid using UF even if technically possible. HEC should be protected by certificates which HF can do very easily. The UF was designed with the thought(assumption) it would read from local storage and forward. The HF and previously intermediate forwarder (which was just HF lite) was designed to receive and forward.

Because of the design intent I am assuming a more robust secure testing would occur at the HF than the UF for any issues.

Re: How can we avoid the scaling issues with on-prem HEC subclusters?

PickleRick — Mon, 16 Dec 2024 21:59:06 GMT

Actually TLS mutual authentication is done by the openssl library and can be configured on an intermediate UF as well (did it myself several times on s2s inputs).

It's just that http input isn't officially supported on UF (any documentation about HEC mentions only Splunk Enterprise or Cloud). So in case anything goes sideways first thing you'll hear from support is "use a HF instead of UF".