I am interested in forwarding syslog and Windows events from a DMZ to Indexers which reside inside our network. We are planning to install universal forwarders both on the syslog and Windows servers, and configure them to forward the events to an intermediate forwarder which will be configured to communicate directly with our Indexer cluster. Our sole intent in doing this is to have only one machine in the DMZ communicating with the indexers. Does the intermediate forwarder need to be a heavy forwarder, or could a universal forwarder be used?
I have this situation also. I used a heavy forwarder (enterprise) because the systems inside the firewalled cloud can't talk to the outside world, and so I needed a deployment server on the DMZ for them to talk to. So that system on the DMZ acts as an intermediate forwarder as well as deployment server. I'm using SSL and encryption for all forwarding connections, and on the DMZ forwarder (inside) that is port 9998, and the same for the indexer it is forwarding to.
I've since read some material that implied there was a way to use the intermediate forwarder as a deployment server proxy, but I have not had time to look into that.
Before data can be made searchable (indexed) it goes through a number of processing pipelines. If you use Heavy weight forwarder (HWF), it can process some of those pipelines for example line breaking. This offload of certain processing pipelines to HWF can reduce load on your indexers.
I would also recommend securing the forwarded data by sending it over SSL to whichever intermediate forwarder you choose, (HWF vs UF).
You can read more about this here. http://docs.splunk.com/Documentation/Splunk/6.1.4/Security/Aboutsecuringdatafromforwarders,Heavy weight forwarder had some advantanges over UF, which include processing of the inputs that it receives. There are some processing pipelines that data needs to go through before it becomes indexable (hence searchable). Heavy weight forwarder can do some of this processing before it sends data over to indexer, for example line breaking of data. This can reduce load on the indexers.
I would also recommend using ssl to get data from the intermediate forwarder that you choose. (universal or Heavy). This will ensure your data is sent securely.