Getting Data In

Will I get duplicate events if I have Multiple Heavy Forwarders (HF) that each have an API connect add-on installed?


Hi guys,

I have a distributed environment in which there are a cluster of indexers and 3 heavy forwarders. Each HF has an add-on/TA which is used to get data by API http connection.

In this configuration, may I have duplicated events at the indexer?

What should I do?

May I install the add-on/TA on only one HF? In this case, I'd have problems with high availability...

Thank you in advance


0 Karma

Path Finder

If multiple servers run the same API calls, you will end up with duplicate data, since they behave like scripted inputs.

A over-engineered solution I can think about, is installing the TA on a SHC, and creating a custom command the will do that API call, then do a scheduled search which calls that command. And since its a SHC, the search will only be called once from one of the search heads, then you HA and no dups.

You can create a SHC from those HWF, providing you have 3 of them.

This will require some work on your side, but it can be a interesting exercise.

Ultra Champion

Hi @lauraG85,

Yes there are possibility of duplicate data in your splunk instance and it totally depends on application from where you are fetching data in Splunk using Add-on. If application stores some checkpoint value, once those data is fetched by any Heavy Forwarder then you'll not end up with duplicate data.

I don't know which Add-ons you are using and what data you are fetching so in general if you do not want to install Splunk Universal Forwarder on server or due to limitation you can't install Splunk UF then you might use HTTP Event Collector (HEC) functionality to send data from application to Splunk server directly and you can setup load balancer to load balance data to multiple Heavy Forwarder for HA functionality.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...