So after months of battling an issue with our indexers dropping connections, we determined that there was a problem with the indexers performing reverse DNS lookups for the connecting servers. To mitigate, we added 'connection_host = none' to the inputs.conf resolving the issue.
If I understand how the host field in the indexed events is populated correctly, with 'connection_host = none' set on the indexers we will now rely on the 'host = ' field in inputs.conf on the UF's. I know this value is automatically populated with the server name when Splunk is first installed, however what happens if a server is renamed? Will it modify the inputs.conf to replace the 'host =' field with the new server name?
It won't. You can change the default host fields name using method described here:
https://answers.splunk.com/answers/154999/how-can-i-change-the-default-hostname-in-splunk.html
With several thousand forwarders it seems that the risk of having an incorrectly named host would be high. Is there a better way to manage this other than reverse DNS lookup?
Any idea what does the following setting in inputs.conf does?
host =
* If set to '$decideOnStartup', will be interpreted as hostname of executing
machine; this will occur on each splunkd startup.