Should I stay away from Windows Event Log when Im setting up custom data inputs?

[sideview]

We're setting up a custom data input and I'm wondering whether it's a bad idea to just write everything to WinEventLog, and then have Splunk index it from there. From the .NET side this seems like a very cheap and simple way to go, whereas setting it up as a scripted input in this case will actually require a bit more work.

But we're concerned that this will be an awful performance bottleneck, or worse that it'll look great for a while and then fail catastrophically under load someday.

It'll be quite a lot of data coming through this path and maybe Windows Event Log is only a good solution if you're dealing with tiny data...

Thanks in advance.

[gkanapathy · Accepted solution]

Windows Event Log inputs work just fine, even under high (AD Server for large domains under high levels of auditing) loads. What doesn't work fine under high volumes is trying to collect Windows Event Logs over WMI.

[southeringtonp]

Can you provide some more detail about the application?

You say you're setting up an input, but then reference .NET - are you working with an existing .NET application and (potentially) writing code?

If so, then the first thing is to choose a logging framework such as log4net or the one provided by the .NET Enterprise library, and then decide where to send the logs in the configuration. Make the decision one of configuration, not code.

If you're not writing code, look to see if such a framework has already been used.

Personally I'd have it send to a TCP or UDP socket, in part to simplify sourcetype assignment. I suspect that the load will be lower, but that it won't really matter until you reach some particular threshold level. As long as it's configurable though, it's not likely to be a major issue since you can easily change your mind later if you decide to go through the Windows Event Log for the time being.

If this isn't what you're looking for at all, can you clarify?