I’m a seasoned Splunk admin and I recently noticed that I'm not aware of any Windows-specific installation best practices for my endpoints. Do these exist? Are there any best practices that apply only to installing Splunk on Windows endpoints?
Said another way, what things specific to Windows, did you wish you knew before installing Splunk on a wide scale?
The Get Windows Data section of the Getting Data In manual exposes the nuance of data collection on Windows and the necessary installation prerequisites. Building on that, this posts calls out or expands on key items.
Universal Forwarder First and foremost, the Initial considerations for deploying Splunk Enterprise on Windows documentation states,
"The most efficient way to gather data from any Windows server is to install universal forwarders on the hosts that you want to gather data. Universal forwarders use limited resources. In some cases, such as Registry monitoring, you must use a forwarder, because you cannot collect Registry data over WMI."
Splunk forwarders versus WMI. Considerations for deciding how to monitor remote Windows data encourages the "use of a universal forwarder to get data in from a remote Windows host. A universal forwarder offers the most types of data sources, provides more detailed data (for example, in performance monitoring metrics), minimizes network overhead, and reduces operational risk and complexity. It is also more scalable than WMI in many cases." See the section Splunk forwarders versus WMI for the trade-offs. Also, WMI rarely works with Splunk apps and solutions, including Splunk premium apps. Lastly, from a security perspective, WMI utilizes a method of access which is considered insecure with well understood exploitation means. For these reasons and more, it is considered a best practice to use a Splunk forwarder installed on remote hosts and avoid WMI for Splunk data collection.
Choose the Windows user Splunk Enterprise should run as. The user that Splunk Enterprise runs as determines what Splunk Enterprise can monitor. See Choose the Windows user Splunk Enterprise should run as within the Splunk Enterprise Installation Manual to learn about the options. If you're not sure, consider using a local system user to start and safeguard against malicious use of the account.
Installation Options. The Splunk® Universal Forwarder documentation has many topics related to installation on Windows. A best practice is to perform the installation a consistent way with limited post installation configuration. This means performing as simple and clean of an installation as possible often only adding post installation configuration for the forwarder to communicate with a deployment server for all configuration. This practice is not Windows specific but what is unique to Windows is installation User Interface. This feature may overshadow the existence of options to Install a Windows universal forwarder from the command line or even Install a Windows universal forwarder remotely with a static configuration. Even more effective is to Make a universal forwarder part of a host image, which is applicable beyond Windows systems.
Next Steps. While the scope of this post is specific to the installation, the next logical step is to get data in. For that, we recommend reviewing the official documentation on Monitoring Windows data with Splunk Enterprise in addition to our post Is it a best practice to use the Splunk Add-on for Microsoft Windows?.
As you might surmise, an operating system agnostic set of best practices is something we'll put together eventually. Until then, all are welcome to add to this Windows-only topic with comments, additions, or adjustments. Of course, if we incorporate your feedback to this post, we'll toss you the karma for your contribution.