Deployment Architecture

Which is better; HEC on heavyforwarder or clustered indexers for Kafka?

splunkreal
Motivator

Hello,

we plan to try Kafka as data collector and we'd like to know if we should keep our HF to receive HEC inputs for Kafka data or send directly to indexers, about 200-300gb per day?

Looks like HF is better for filtering before indexing.

Thanks.

 

* If this helps, please upvote or accept solution if it solved *
Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

An HF is just an indexer that doesn't store data so it's no better or worse for filtering than an indexer.

A single HF is a single point of failure for your Kafka data.  It also can lead to poor data distribution which will negatively affect search performance.  The solution, of course, is to have multiple HFs, but if you're going to round-robin among HFs, why not just round-robin among indexers?

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

An HF is just an indexer that doesn't store data so it's no better or worse for filtering than an indexer.

A single HF is a single point of failure for your Kafka data.  It also can lead to poor data distribution which will negatively affect search performance.  The solution, of course, is to have multiple HFs, but if you're going to round-robin among HFs, why not just round-robin among indexers?

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Monitoring Postgres with OpenTelemetry

Behind every business-critical application, you’ll find databases. These behind-the-scenes stores power ...

Mastering Synthetic Browser Testing: Pro Tips to Keep Your Web App Running Smoothly

To start, if you're new to synthetic monitoring, I recommend exploring this synthetic monitoring overview. In ...

Splunk Edge Processor | Popular Use Cases to Get Started with Edge Processor

Splunk Edge Processor offers more efficient, flexible data transformation – helping you reduce noise, control ...