Getting Data In

Will I get duplicate events if I have Multiple Heavy Forwarders (HF) that each have an API connect add-on installed?

lauraG85
Engager

Hi guys,

I have a distributed environment in which there are a cluster of indexers and 3 heavy forwarders. Each HF has an add-on/TA which is used to get data by API http connection.

In this configuration, may I have duplicated events at the indexer?

What should I do?

May I install the add-on/TA on only one HF? In this case, I'd have problems with high availability...

Thank you in advance

laura

0 Karma

rabbidroid
Path Finder

If multiple servers run the same API calls, you will end up with duplicate data, since they behave like scripted inputs.

A over-engineered solution I can think about, is installing the TA on a SHC, and creating a custom command the will do that API call, then do a scheduled search which calls that command. And since its a SHC, the search will only be called once from one of the search heads, then you HA and no dups.

You can create a SHC from those HWF, providing you have 3 of them.

This will require some work on your side, but it can be a interesting exercise.

harsmarvania57
Ultra Champion

Hi @lauraG85,

Yes there are possibility of duplicate data in your splunk instance and it totally depends on application from where you are fetching data in Splunk using Add-on. If application stores some checkpoint value, once those data is fetched by any Heavy Forwarder then you'll not end up with duplicate data.

I don't know which Add-ons you are using and what data you are fetching so in general if you do not want to install Splunk Universal Forwarder on server or due to limitation you can't install Splunk UF then you might use HTTP Event Collector (HEC) functionality to send data from application to Splunk server directly and you can setup load balancer to load balance data to multiple Heavy Forwarder for HA functionality.

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...