How can we avoid the scaling issues with on-prem HEC subclusters?

[danielbb]

On Splunk cloud, we can receive HEC ingestion directly to the cloud whereas on-prem we install distinct subclusters for HEC and struggle to scale them up with multiple down-times cases. I wonder whether it's possible for an on-prem installation to send HEC data directly to the indexers. 

[danielbb]

I love of the idea of the - HEC inputs directly on indexers (and have an LB in front of them) !

[isoutamo]

You should look this https://docs.splunk.com/Documentation/SVA/current/Architectures/TopologyGuidance
It contains preferred Splunk architecture layouts.

You should remember that if you have lot of HEC inputs and you need to update/add those regularly this impacts your indexers it you are using those instead of HFs as HEC inputs.
For that reason I personally prefer to use couple of HFs with LB as a HEC cluster instead of configure those directly into indexers.

Here is some instructions how to tune HEC.
- https://community.splunk.com/t5/Getting-Data-In/What-are-the-best-HEC-perf-tuning-configs/m-p/601629
- https://community.splunk.com/t5/Getting-Data-In/Can-we-have-fewer-Heavy-Forwarders-than-Indexers/m-p...

[danielbb]

Thank you all, since I have already the HEC cluster and it is with HFs, wouldn't it be better to have UFs  since my understanding is that data flows quicker through UFs? and we don't process the data at all at this HEC cluster level.

[PickleRick]

Officially HEC isn't supported on UF. I read several times that it works but never actually got to testing it myself.

[dural_yyz]

I agree about recommending to avoid using UF even if technically possible.  HEC should be protected by certificates which HF can do very easily.  The UF was designed with the thought(assumption) it would read from local storage and forward. The HF and previously intermediate forwarder (which was just HF lite) was designed to receive and forward.

Because of the design intent I am assuming a more robust secure testing would occur at the HF than the UF for any issues.

[PickleRick]

Actually TLS mutual authentication is done by the openssl library and can be configured on an intermediate UF as well (did it myself several times on s2s inputs).

It's just that http input isn't officially supported on UF (any documentation about HEC mentions only Splunk Enterprise or Cloud). So in case anything goes sideways first thing you'll hear from support is "use a HF instead of UF".

[PickleRick]

+1 on that. While an input/parsing HF layer isn't exactly SVA I like this approach because it isolates inputs and index-time parsing settings from the indexers. So the only maintenance you need to do on the indexers is strictly about indexes and storage.

[dural_yyz]

In a past environment I ran isolated HF's specifically for HEC with no other purpose.  I was able to tune them up on HEC processing cause there was no conflicting use cases for the compute power.  I had 2 sites with 2 HF's per site all acting in full HA behind site LB and local LB configurations.  The HF's were pointed to the Deployment Servers so the HEC inputs and config could be updated in a central location with auto distribution.

To size up I would only have to stand up another HF in either location one at a time or in bulk.  Have them point to the DS and confirm the local logs were found at the indexers.  Then I could add the new server addresses to the server class for the HEC inputs/config to upload.  Then update the LB to include the new server(s) in the pool for that particular site.

Very convenient and not difficult once setup - although not for novice users though.  Plugging an HF to a DS can come with issues if not 100% aware of how apps are named.

[PickleRick]

I'm not sure what you mean by "subcluster". You can have HEC inputs directly on indexers (and have an LB in front of them) or can have a farm of HFs with HECs sending to the cluster.