best practise: remote log servers

[sayz]

hi;

we have 7 remote log servers which we are sending all of our logs from approximately 400 different servers(apache, dns, ldap etc...).

so, I want to ask you that what is the best practice to forward our logs to the splunk cloud? installing forwarders on 7 remote log servers or on the servers we have? and why? could you explain to me?

thanks,

[thomasb42]

I suppose with "Splunk Cloud" you're referring to the Splunk SaaS service. Then your data leaves your network and passes on to the Internet. I would carefully consider about the security implications. If every server has a forwarder connecting directly to the cloud, you need to open up a lot of holes in your firewall and it's difficult to control the traffic flow. Best practice (from my point of view) is having a forwarder (or pair of, if redundancy matters) at the edge of your network (maybe in a dedicated DMZ zone), used as intermediate between your network and the cloud. Allows you to isolate the servers in your network from the Internet (and vice versa). I wouldn't allow internal machines to directly open connections out of the network.

Having forwarders on each server might be more reliable. As an example, transporting data via forwarders is more reliable then sending them over UDP syslog or polling them via WMI. Thus I would consider deploying universal forwarders on each server, let them forward to the heavy (intermediate) forwarders at the edge of your network and send to the cloud from there. The intermediates could also be used as deployment server to keep the configuration of all the universal forwarders in sync.

[SierraX]

Hi,

I would prefer the usage of 7 forwarders. Because when something happen on Splunk Server side what block incoming data for a while the for splunk forwarder hold in cache and send when the connection is unblocked.
Also with a restart of a Splunk forwarder it looks where to continue reading ... not sure every log-server do this instead of sending the whole files again and running against a possible CRC problem or produce double events.

[sayz]

Well the issue here isn't about using forward. i already explained that ill be usin forwarder. its about how u use the forwarder, meaning not about the comprassion of data but about where to put the forwarder? on a dedicated 7 log servers or every single server that we need to collect data should have also a forwarder deployed on them so that they all together send the logs to splunk cloud? the main question is if there is any with hands on experience on this subject and can lead me the right direction?

[TStrauch]

In addition with forwarders you can compress your data before sending it to the indexer. You can encrypt your connection between the splunk instances, you can specify sourcetypes at the forwarder site to get some load from the indexers and so on. There are many benefits of using the forwarders.

You just can say. if you can use them. Use them.