Multiple Heavy Forwarders for data Receiving.

[SagarSplunk]

Hi All,

Can we have multiple heavy forwarders to filter and forward data to multiple indexers in a environment.

e.g. 20-40 UF ==> 2 Heavy Forwrders ==> 3 Indexers ==> 3 Searchheads

We have 20-40 universal forwarders installed in a environment and all UF has different log files as data sources. we have below requirments.
1) To Filter unnecessary data from data sources
2) Heavy Forwarders will only filter data no indexing will happen at Heavy Forwarder level.
3) High Availability for Heavy Forwarders so we introduced 2 HF.

NOTE:-
1) Above are details for DC1 Site same setup is there for DC2 to achieve HA.

==================================================================
Questions:-
1) Do we really need 2 HF at each site to achive HA
2) Will there be any data duplication at indexer level if we use 2 HF at each site.
3) How we can achive High availability at Heavy Forwarders layer at each site will there be any fail-over mechanism required.

Thanks

[SagarSplunk]

Hi Adonio,

Thank you for response, below are Reponses for your queries/questions/suggestions.
1. We have data center1 and data center 2 in our environment for High availability of solution.
2. We have Splunk Search head cluster, indexers cluster, deployment server and license manager and all these components are in cloud environment.
3. Heavy Forwarders and Universal Forwarders will be installed on premises and will have to configure HA for HF.

4. We are using Heavy Forwader because we want to filter some unwanted data and send only data to indexer which customer is interested in.

Questions:-
a. Is it possible to provide HA at HF layer
b. will there be any data duplication happen if we introduce 2 HF
c. 1 HF is enough for filtering

[adonio]

Hello @SagarSplunk,
i answered your questions in my answer above. also look at @skalliger comment, this is the reason i said "sort of HA" or "HA" in double quotes.
so again, for question a: the answer is like above but if you want a yes no answer, then no it is
for question b: the answer is like above, no
for question c: on its surface, it looks like 1 HF is enough for 20-40 forwards.
will keep on pushing on the necessity of HF, will not use unless i must
hope it helps

[SagarSplunk]

Got it,Thank you for the response.

[skalliger]

Heavy Forwarders do not have a high availability/failover feature.

[SagarSplunk]

Thank you for confirming. Is there any third party tool which can take care of this is Splunk HF don't have this capability

[skalliger]

Yes, a third-party loadbalancer. Look at this blog post for the latest architecture Splunk presented: http://dev.splunk.com/view/event-collector/SP-CAAAE73
So, what you could do, is, place a third-party loadbalancer in front of your indexers, which get the data via HTTP event collection (HEC).

I haven't set up a HEC environment myself yet, so I can't really give any tips about it.

Skalli

[SagarSplunk]

Thanks a lot for the response, Appreciate it.

[adonio]

hello there:
first i will recommend to avoid HF unless you have to use it, for example, you will use DB Connect.
1. with that said, you can achieve sort of HA if you have the same outputs on HF and the forwarders are configured to load balance across the HF. however, you did not mention whether your search heads are clustered or indexers are clustered so I am not sure exactly what you are try to protect from? is it a single failure of a server?
2. there will not be any data duplication if all is set correctly
3. as explained above, you can achieve some "HA" and if all HF are configured exactly the same, no fail-over mechanism required
would like again to recommend against using the HF unless you really have to. it adds another layer to manage and maintain
hope it helps