https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166320#M33721 <P>I recently had this discussion, but it was more around using regional heavy forwarders that feed in to a central indexer. We had existing heavy forwarders for syslog collection, so had to decide whether or not to send universal forwarder data towards them, or direct to the indexer.</P> <P>Pros of using HF / cons of not using HF:</P> <OL> <LI>Traffic can be uncompressed / unencrypted towards the HF, saving CPU, the HF can then compress and/or encrypt it for bandwidth and security, as it's got dedicated CPU</LI> <LI>The HF can drop and route events</LI> <LI>The HF can re-write data, such as credit cards, before sending it to the IDX</LI> <LI>The HF can cook the data before sending it to the IDX</LI> </OL> <P>Pros of not using HF / cons of using HF:</P> <OL> <LI>Each UF has the same output configuration, making it easier to manage (deployment server mitigates this)</LI> <LI>If a HF goes down (planned or unplanned), logs from the UFs still arrive</LI> <LI>The general philosophy of Splunk seems to be it's better to go direct</LI> <LI>If you have 1000 UFs doing auto load-balancing between two IDXs, the split will be very close to 50/50 as they all switch around. If you have 4 or 5 HFs, it's entirely possible you could end up with all of them pointing to one IDX and hammering it before they switch out.</LI> </OL> <P>Can it be done? Yes. Should it? That's entirely up to you, and hopefully this helps you ask the right questions for your scenario.</P> Thu, 05 Dec 2013 08:36:23 GMT sciurus 2013-12-05T08:36:23Z https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166318#M33719 <P>Hi,</P> <P>As we begin to roll-out hundreds and thousands of universal forwarders, I was wondering if it made sense to create a layer of heavy forwarders, to take some load off the indexers. Has anyone done this? Does it make sense? </P> Wed, 04 Dec 2013 15:24:33 GMT https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166318#M33719 a212830 2013-12-04T15:24:33Z https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166319#M33720 <P>What about having multiple indexers to share the load? <A href="http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Setuploadbalancingd">http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Setuploadbalancingd</A></P> Wed, 04 Dec 2013 16:31:05 GMT https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166319#M33720 somesoni2 2013-12-04T16:31:05Z https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166320#M33721 <P>I recently had this discussion, but it was more around using regional heavy forwarders that feed in to a central indexer. We had existing heavy forwarders for syslog collection, so had to decide whether or not to send universal forwarder data towards them, or direct to the indexer.</P> <P>Pros of using HF / cons of not using HF:</P> <OL> <LI>Traffic can be uncompressed / unencrypted towards the HF, saving CPU, the HF can then compress and/or encrypt it for bandwidth and security, as it's got dedicated CPU</LI> <LI>The HF can drop and route events</LI> <LI>The HF can re-write data, such as credit cards, before sending it to the IDX</LI> <LI>The HF can cook the data before sending it to the IDX</LI> </OL> <P>Pros of not using HF / cons of using HF:</P> <OL> <LI>Each UF has the same output configuration, making it easier to manage (deployment server mitigates this)</LI> <LI>If a HF goes down (planned or unplanned), logs from the UFs still arrive</LI> <LI>The general philosophy of Splunk seems to be it's better to go direct</LI> <LI>If you have 1000 UFs doing auto load-balancing between two IDXs, the split will be very close to 50/50 as they all switch around. If you have 4 or 5 HFs, it's entirely possible you could end up with all of them pointing to one IDX and hammering it before they switch out.</LI> </OL> <P>Can it be done? Yes. Should it? That's entirely up to you, and hopefully this helps you ask the right questions for your scenario.</P> Thu, 05 Dec 2013 08:36:23 GMT https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166320#M33721 sciurus 2013-12-05T08:36:23Z https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166321#M33722 <P>I will have multiple indexers, but I also want to do whatever is possible to reduce the load on the indexers.</P> Thu, 05 Dec 2013 11:39:09 GMT https://community.splunk.com/t5/Getting-Data-In/intermediary-heavy-forwarder-tier/m-p/166321#M33722 a212830 2013-12-05T11:39:09Z