What is the best practice to capture data from our *nix servers? Install the Splunk forwarder agent and the Splunk for Unix app which feeds directly to our indexers? I thought a best practice was to have everything syslog-ed before being indexed.
If we only use remote syslog on our servers (not having the Splunk forwarder agent on our servers), I'm assuming we won't get the metrics that the Splunk for Unix app polls for.
Typically, we would suggest a Universal Forwarder on the *nix server. This removes any latency that might be introduced by forwarding the syslogs to a syslog server, and also allows you to collect performance data and other logs from the server. The recommendation to use a syslog server is generally for devices or appliances that don't allow for the installation of a forwarder. My rule of thumb is to collect the data at the source whenever possible.
Also prevents difficulties with missing timezone information. Hundreds of servers with arbitrary time zone config logging to a central syslog server can be a serious nightmare to deal with if the servers are not including timezone in their syslog message (which in default basic syslog forwarding format is not included).