What is the best way to collect Windows Event Log data from 900+ computers?

[fdarrigo]

What is the best way to collect System and Security Windows Event Logs from my 900+ computers?

Option1
Install the Universal Forwarder on 900 computers. I already have an SCCM installation package, so I am not concerned about UF deployment, but I am concerned about adding 900 additional sources to Splunk.

Option 2
I can configure central WinEvent logging on a server and install the UF on that server.

Option 3
There are tools which convert WinEvent to syslog format, which can then be sent to my syslog server.

How have you solved this problem?
Are there other options I haven't considered?

[sloshburch]

Option 1 and I just posted more details that might be effective for your needs How do I collect basic Windows OS Event Log data from my Windows systems?

[tskinnerivsec]

If you wanted to send to a central server first, before sending into splunk for indexing, a better approach would be to still use the Splunk universal forwarders to send data, but install splunk on that central server and configure it as a Heavy or Intermediate forwarder. It would centralize the collection of your windows log data, and it can pre-parse any data you want before forwarding on to indexers. You would be able to use internal splunk logs on that Heavy Forwarder to validate log data is being sent. Installing UFs is very trivial and does not add much complexity or load to servers. Configuring splunk deployment server to manage configs of UFs makes managing them as a group pretty easy.

[nychawk]

Great point on the pre-parsing ability afforded, this would certainly help decrease the noise. And, while I agree that installing UF's is trivial, monitoring them is not; although I will investigate your suggestion to use internal Splunk logs to validate log data, or the absence of.

Would love to hear feedback on woodcocks idea though...

Thank you,

-mi

[woodcock]

Check this out; Splunk has made a major change to end this problem once and for all in 6.2:

http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/

[fdarrigo]

Thanks for the link. I'll try it out

[woodcock]

If it works as well as it should (I have not personally had a chance to try it), be sure to come back here and update us and then "Accept" and answer to close out the question.

[tskinnerivsec]

You definitely lose some reliability and fidelity if you try to convert the log data to syslog then send it. You need to determine how much index volume you have available and make sure the load won't take you over the limit. Installing a UF along with the Splunk_TA_windows with the inputs that you want enabled is the easiest and most recommended, especially if you are using the deployment server to manage your UF splunk configs. The UF has a small footprint, so shouldn't present an issue to the end clients. Splunk scales to whatever load you throw at it, whether it be from syslog or UF feeds. The only challenges (and this would be a challenge no matter what method you use to send the event data) is that if you have a distributed environment, you need to assure all of the clear network paths from the UFs to Indexers (or heavy forwarders) to send the event data.

[nychawk]

I too am building out a new WinEvent logging design, and have considered the three solutions discussed above by fdarrigo.

For me, option 1, UF's everywhere, was not an option; I felt monitoring each one would only add more complexity, and potential single point of failure. Besides, I would have to verify a UF install was properly done for each new server, a major pain.

Option 2, centralized WinEvent server is thus far my favorite choice, and requires UF installed on the least number of nodes as compared to option 1.

As discussed in option 3, I've used WinEvent to syslog converters in the past with great success, however I wasn't as concerned then with log fidelity as I am now.

My goal is to deploy option 2, centralized WinEvent log server, and have the central server retain it's own logs for whatever my disk limitations will allow, most likely 4-6 months. Since the data will be delivered into Splunk, I can retain there even longer. The only issue I see here is validating each sender to my central WinEvent store is actually sending, however this is off-topic; but would love to hear if others have any good ideas.

With regards to woodcock's reply, does converting to XML raise any questions to log fidelity? Also, are their any current dashboards that can consume XML to render reports for nefarious login activity, and/or changes to AD accounts and groups?

Another option might be to deploy a central WinEvent logging server, and deploy a WinEvent to syslog converter on that versus installing a UF; not sure what added benefit, if any, this offers.

Kind regards,

-mi