Is it recommended to install a universal forwarder on thousands of workstations or on a few dedicated syslog/Windows Event Collector servers?

[flee]

Hello.

We have a project that needs to forward Windows events or text files from approximately 6000 Windows workstations. Would it be advisable to install Universal Forwarder (UF) on each of the 6000 workstations or have only a few dedicated syslog collector or Windows Event collector servers where the UF would be installed? Any deployment or on-going maintenance challenges to manage 6000 UFs from the deployment server?

Thanks in advance for your advice!

[itrimble1]

What solution did you end up going with ? The UF's or the WEC ?

[hochit]

In my opinion, how many indexers serving those forwarders is also critical. It could overload indexer's network interface or it's disk.

[javiergn · Accepted solution]

Splunk recommendation is to always install the UF locally whenever possible.
That means 6000 UFs in your case.

PROS:

• Flexibility. You might want to filter certain things on very specific UFs or just collect certain logs. You might want to be able to read very specific log files you never thought about before and if you have a local UF, that's perfectly doable
• Resiliency. Do not put all your eggs in one basket. If your dedicated syslog or Windows Event collector are down or under maintenance, then you are blind
• Performance. UF footprint is usually tiny. If you try to read a huge amount of logs from one place it might struggle or have a major impact on your dedicated server
• Failover. If you want half or your UFs to forward logs to Indexer 1 and 2, and the other half to forward to Indexer 3 and 4. That's all perfectly doable. Combinations are unlimited.
• SID translation for Event Logs. Careful if you are going to use SID translation with your event logs as this is going to hit your global catalog unless you specify a default name server. But even if you do that, if you try to do SID translation from one dedicated server that contains logs from 6000 workstations, this might cause problems in your domain controllers. And I'm talking from experience here. Unless there's some sort of DNS round robin in place or Splunk has implemented SID caching on 6.3, this is going to be a headache.

CONS:

• Deploying and maintaining 6000 UFs is obviously more complex and it'll require a lot of configuration
• Your Deployment Server might struggle with 6000 UFs so make sure the agents are not calling home every 30 seconds and increase this time to several minutes
• Deploying configuration files to such amount of UFs can take 30 minutes easily or even more depending on your network bandwidth
• Your deployment server has to be properly secured in order to have a proper change control. Keep in mind your UFs will probably run with some privileges not all users have so you could elevate your access easily by asking one of those agents to carry out certain tasks you wouldn't normally be able to do with your user account. There are ways to secure this but I don't want to extend here.
• Network access is required between those UFs and the deployment server. You can configure a multi-tier deployment server model, but this is even more complicated to maintain, although it's perfectly doable.
• Do not run your Deployment Server on Windows if you are going to deploy configuration files to Linux. Do it the other way around, otherwise you are going to end up with a big mess in on those Linux UFs because the Windows deployment will break your permission model. I don't know if this has been fixed on 6.3 by the way.

If you are too worried about the Deployment Server and you are already using tools such as Puppet to manage your configuration files, stick to it and use it for your Splunk forwarders too.

Hope this helps,
J

[flee]

Thanks so much Javiergn for your detailed advice!

[stephanefotso]

Hello.
In fact, you must install a Universal Forwarder (UF) on each of the 6000 workstations to get logs of each machine. But if you have a Windows Event Collectort Server, which collect logs of all the 6000 machines, you can install your UF on this machine if you like. For more informations about forwarding data, read here: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Forwarding/Universalforwarderdeploymentoverview

Thanks.

[flee]

Thanks stephanefotso!