Do I need to install Universal Forwarders to all the remote servers I want to monitor?
It is not mandatory to use a UF as there are several potential topologies to push and pull data into your Splunk Indexer(s).
It really boils down to your particular data source & architectural requirements.
1) UF on each server
2) Dedicated UF instance where a UF can not be installed ie: to receive and load balance syslog feeds from network devices or to open up privleged ports
3) No use of a UF...raw feeds to TCP/UDP socket inputs on the Indexers, syslog, scripted inputs that remotely poll your sources, rsynced log files to your Indexers etc...
Universal Forwarders are certainly a best practice to data collection and can further augment your architecture with load balancing, data cloning, compression, encryption, indexer acknowledgement,throttling, centrally configurable using Deployment Manager etc... I certainly try to use a UF wherever it is possible, but if not possible, Splunk has many other avenues for Indexing your data.
This should help :
http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor
Here's a scenario:
I have 50+ Win Server in LA and I need to monitor those servers here in Virginia. I need to install the SPLUNK Indexer here in VA. Do I need to install the UF to all 50+ servers so it will forward data here in VA? does that mean all 50+ servers will have their own connection when they send data here in VA? OR do I need to just have one UF in LA that will collect all the data from all 50+ servers and then that one UF will send that data here in VA?
Is the UF some kind of agent installed locally on the remote server that collects the data locally and then forward it to the Indexer?
Yes. Forwarders (Splunk agents) allow you to install a lightweight version of Splunk on any number of distributed sources to send data to a central Splunk indexer. A Splunk server running on any supported OS platform can forward data to another Splunk instance (as well as to other systems) in real time. This allows data gathered on one Splunk host in a specific environment to be sent to another Splunk instance for indexing and search.
It is not mandatory to use a UF as there are several potential topologies to push and pull data into your Splunk Indexer(s).
It really boils down to your particular data source & architectural requirements.
1) UF on each server
2) Dedicated UF instance where a UF can not be installed ie: to receive and load balance syslog feeds from network devices or to open up privleged ports
3) No use of a UF...raw feeds to TCP/UDP socket inputs on the Indexers, syslog, scripted inputs that remotely poll your sources, rsynced log files to your Indexers etc...
Universal Forwarders are certainly a best practice to data collection and can further augment your architecture with load balancing, data cloning, compression, encryption, indexer acknowledgement,throttling, centrally configurable using Deployment Manager etc... I certainly try to use a UF wherever it is possible, but if not possible, Splunk has many other avenues for Indexing your data.
This should help :
http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor
You should deploy a forwarder for each remote server that's sending data to an indexer. See http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Aboutforwardingandreceivingdata for information about deployment topology and http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Deploymentoverview for information about deploying the universal forwarder in particular.