Getting Data In

Will I get duplicate events if I have Multiple Heavy Forwarders (HF) that each have an API connect add-on installed?

lauraG85
Engager

Hi guys,

I have a distributed environment in which there are a cluster of indexers and 3 heavy forwarders. Each HF has an add-on/TA which is used to get data by API http connection.

In this configuration, may I have duplicated events at the indexer?

What should I do?

May I install the add-on/TA on only one HF? In this case, I'd have problems with high availability...

Thank you in advance

laura

0 Karma

rabbidroid
SplunkTrust
SplunkTrust

If multiple servers run the same API calls, you will end up with duplicate data, since they behave like scripted inputs.

A over-engineered solution I can think about, is installing the TA on a SHC, and creating a custom command the will do that API call, then do a scheduled search which calls that command. And since its a SHC, the search will only be called once from one of the search heads, then you HA and no dups.

You can create a SHC from those HWF, providing you have 3 of them.

This will require some work on your side, but it can be a interesting exercise.

harsmarvania57
SplunkTrust
SplunkTrust

Hi @lauraG85,

Yes there are possibility of duplicate data in your splunk instance and it totally depends on application from where you are fetching data in Splunk using Add-on. If application stores some checkpoint value, once those data is fetched by any Heavy Forwarder then you'll not end up with duplicate data.

I don't know which Add-ons you are using and what data you are fetching so in general if you do not want to install Splunk Universal Forwarder on server or due to limitation you can't install Splunk UF then you might use HTTP Event Collector (HEC) functionality to send data from application to Splunk server directly and you can setup load balancer to load balance data to multiple Heavy Forwarder for HA functionality.

0 Karma

Tune In & Win!

Don't miss out on your
chance to take home free
prizes by helping our players
save the Splunk Cloudom!

Dungeons & Data
Monsters: Splunk O11y
Day Editions Games
stream live:
5/4 at 6:30pm PST
5/5 at 7:00pm PST
on