How should I go about using one forwarder for all servers?

[thijsvl]

Hi Splunk community,

I want to have a single forwarder for every on-premise domain controller in my network, instead of installing a universal forwarder on every domain controller.

How do I go about doing this? And what kind of forwarder should I use?

[Richfez · Accepted solution]

If you don't want to install the Splunk Universal Forwarder (UF) on every Domain Controller (DC) you have, then you have a couple of options.

1) Use Windows built-in event log forwarding to have all the DCs forward their event logs to one machine, then install the UF on that one machine and use it to pull in all the logs from the DCs via those forwarded events. This has the advantage of having only one Splunk UF installed. It has several disadvantages —reliance on the event log forwarding and it properly continuing to work (I've found it problematic at times); a potentially HUGE load placed on the receiver of those events; and introducing two separate systems to maintain (event log forwarding and also the UF). And also, you'll STILL need the UF on one DC if you want to collect actual Active Directory (AD) logs (not local security logs, but the AD side of it).

2) Use Windows Management Instrumentation (WMI) to collect the logs from each DC. This has the advantage of centralized administration (you can admin this whole mess from Splunk), but the disadvantages are even higher than in #1 above. The performance is pretty bad. You may get lag on inputs, depending on how busy your DCs are you may need as many as one WMI receiver for each half dozen DCs, and you'll increase the load on the DCs quite a bit because they're now responding to a LOT of extra WMI requests. And you'll STILL need the UF on one of them to collect the actual AD logs.

3) Install the UF — either manually or via SCCM or whatever — on all the DCs. This is the highest performing option (least load on the Splunk servers, least load on the DCs themselves, most likely to "keep up with the data" best). It's also able to be centrally managed from Splunk (except the actual install of the UF itself — but all its configs are manageable via the Deployment Server). It has the least impact on a WAN (compression), and arguably, this is the one that's best supported — there are millions of these installs out in the wild. It's 'the way to do it'. The disadvantages are that, once or twice a year, you'll have to update them all. As a data point - out of 300+ UFs at $job-1, I'd update them via script (search for it, it's not hard) with a silent install/update every 6 months or so, and I think I had to manually bang on perhaps only 3 or 4 in the past 3 years. And that "manual fixing" usually only took about 5 minutes to just do a manual uninstall of the old version and try the script again.

So I'm not sure what exact pain point you are expecting but I can make some recommendations about which to use depending on exactly why you are worried about it

• If it's simply that "corporate won't approve that app on the DCs until it's been vetted" then ... if you have someone brave enough to try to get the event log forwarding working properly (which sometimes it "just works", so maybe this is fine) then I'd probably go that route.
  
• If it's a relatively small environment (under maybe a dozen DCs) and corporate won't let you install software on them, then maybe WMI will be your best option.
• If you are pulling this data over a WAN, especially if it's not 100% reliable, then just install the UFs. All other methods will be painful
• If it's that you are worried about maintaining it, well — script the installs or use SCCM or something. It doesn't generally need updating every minor revision unless you are having a problem (do try to stick with the same major version though, but even in that case a late 6.x UF works fine with a 7.x Splunk, unless you are doing a 7.x thing specifically)

I hope this helps!

[kmorris_splunk]

I have seen some of my customers use Windows Event Forwarding to send the logs to a single Windows box, where you would have a Universal Forwarder. I don't know how performant this is though, compared to running a forwarder on each DC, which would be the optimal solution.

The Universal Forwarder is very lightweight and uses minimal resources if configured properly, which makes it the preferred way to collect the logs.