forwarding windows events and performance

[klee310]

Hi all, I'm testing a setup in which there are two Windows servers. Both Splunk instances also have the Windows app installed and setup. However in this scenario, I want one of the servers (the smaller one) to only forward the Windows events (and performance data) to the second (primary) server. And I want to be able to report on the performance data of both servers from my primary server.

I have figured out how to setup; and have setup the forwarding of data from my smaller server to my primary server (over port 9997). Also, I have turned off all scheduled searches on my smaller server, since I won't be keeping a local copy of data before forwarding.

Now, I can see all of the data related to my smaller server on the primary server. But then, I realized there is something called Remote Performance Monitoring (on the Manager/Inputs page). So I go in and set that up as well. In short, I simply appended the remote host-name to all of the existing saved configuration, for example, CPUTime used to be specified only for localhost, now it is specified for both localhost as well as the hostname to the smaller-server.

Now the question is: are my events coming in from the smaller server going to be duplicated/redundant? Should I use one method over the other? What if my the smaller server wasn't running Windows (ex. Linux, AIX, etc.)?

Any help would be greatly appreciated.

[piebob · Accepted solution]

what i think you're asking is if you set up an input via Splunk Web on the primary instance, and also send the same data to the primary instance via a forwarder on the smaller server, will you get duplicate data? i'd assume that yes, you would--you only need to do one or the other.

with respect to which way is better--in general, Splunk recommends using a forwarder rather than the remote polling option on Windows, for these reasons:

"For identical collecting of local Event Logs and flat files, a forwarder requires less CPU and performs basic pre-compression of the data in an effort to reduce network overhead. It is more memory intensive, however, mostly owing to the additional data source input options available.

If the forwarder is configured to run as the Local System user, the authentication requirements for the local machine are eliminated, as that account has full access to the local machine.

Remote polling over WMI is more CPU intensive on the target machine for the same set of data (either remote Event Logs or remote performance data), and is more network intensive overall. It requires that Splunk runs as a user with explicit access to these data sources."

(from
http://www.splunk.com/base/Documentation/latest/Data/ConsiderationsfordecidinghowtomonitorWindowsdat... )

if your smaller server were running a non-Windows OS, you can make the same choice (forwarder or remote input), although my assumption is that it becomes less about performance concerns and more about ease of management. if you have just a few inputs, it's probably easier to set them up from the primary splunk instance, and not worry about installing anything on the other box at all. it depends on your preference and requirements.

[piebob]

right, all the data will get sent to the primary instance, and the summarization (assuming it is defined there) will occur there instead. you just have to make sure that you configure the forwarder to forward the data you want to the primary instance.

[klee310]

hi piebob, thanks for the reply. To further clarify, is it save to assume that all the data generated on the smaller server (by the Windows app) will be forwarded to the primary instance, save the summary information generated on the smaller server?

In this case, is it safe for me to go one step further and disable all scheduled searches (summary) on the smaller server to decrease CPU usage, since all that raw data being forwarded to the primary instance will be summarized there instead?