we have 2 newly buildup heavy forwarders in our splunk environment, instead of having syslog-ng on separate dedicated servers, we thought of download/install syslog-ng on one of our Heavy Forwarder. is this recommendable? if yes, can someone please send me if there is any documentation??
It is completely fine if you install Syslog-ng on your Heavy Forwarder. There are advantages to this. I am enlisting a few points below please consider them
Heavy Forwarders need higher resources as compared to Universal Forwarders. Please make sure you have enough resources.
As with Heavy Forwarder, you might not be indexing all the data that is received. A plain HF with no indexing requires less storage. But when we switch to Syslog-ng, the storage requirements increase based on the amount of data we ingest.
We need to take care of the log rotation, in case we have a huge volume of data coming to the Syslog server. If the disk gets full, Syslog server might stop working and we lose in-flight data. Old and Indexed data should be removed / archived on time.
Using Syslog server, it is more convenient to manage data before indexing it in Splunk.