@jkat5. I appreciate your enthusiasm for Splunk. I don't share it in the least. That light bulb lit up last week and I cursed it and had to replace it several times before it would stop flickering. No big data analytics here. I find them as exciting as dirt if I have to use a product that requires this level of commitment and expertise to deploy and manage to perform the simplest task effectively. This is clearly a product that requires mastery to perform a simple task. It's incredibly powerful and configurable and you can monitor anything if you're willing to pay the price of entry. Perfectly fine if that's your goal.
I've already concluded Splunk is not my thing. As Zathras said, "This is the wrong tool.". I concluded that when I was given this project and skimmed the documentation for installing the App for Windows Infrastructure. I knew I was in trouble I saw the Splunk certified consultant projected ~$10k of professional services to deploy a new instance that supported what we wanted logged. It turned out to be even more painful than I anticipated once the ugly truth that our indexing costs would quintuple unless I could find a way to Tame the Security Logs. That's why I'm here now.
I don't have the time to master a whole framework and paradigm just to archive logs. I inherited this really expensive Syslog server because somebody read how powerful it was and had all these "Apps" available for it. It was easy to set it up as a Syslog server without a sweat. Once a colleague tried installing the App for Exchange, but quickly abandoned it up after learning how much manual configuration was involved. I had hoped to be able to install the App for Windows Infrastructure, point my Windows servers at it and go on my merry way. Instead I have my daily indexing mushroom from 10 GB /day ingesting all of our Cisco syslog data from hundreds of switches and several firewalls to 80 GB yesterday ingesting security logs from 10 Domain Controllers. I'm currently over 12 GB today after throwing everything I could find to squelch the flow. That puts us well over our recently upgraded license.
I have SCOM to monitor Widows infrastructure. I've had to rewrite a few monitors where M$ programmers were lazy to make things work properly, but otherwise it was install and configure. Most management packs, the equivalent to Splunk Apps work on install. All dependencies are managed by SCOM.
I have customized my inputs.conf file so only the security log is forwarded (see below). Had to, because there was no set that restriction through the GUI. I've also blacklisted a couple events that are known to be noisy. I've even applied some event re-writes on a couple events (Whoo hoo, welcome to .props and .transforms). I've applied tweaks identified in the following blogs, which I found in my attempt to bring this under control. Splunk Support doesn't seem to be aware of these, though two are from an internal blog.
http://blogs.splunk.com/2012/09/21/the-splunk-app-for-active-directory-and-how-i-tamed-the-security-log/
http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/
https://runals.blogspot.com/2014/07/taming-verbose-windows-logs-in-splunk.html?showComment=1462985343336#c4560143038727613202
[WinEventLog://Security]
disabled = 0
suppress_text = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
index = wineventlog
renderXml= 1
The learning curve for Splunk is about as steep as I've ever encountered. The documentation is a horrible hodgepodge of disjointed documents that is about the worst I can remember trying to decipher. Examples are horribly simple and generic with a clear bias on Syslog and *nix use cases and assume *nix installations in most cases. The documentation for the Splunk App for Windows Infrastructure is the Documentation for the Splunk App for Exchange with the name changed. No one even bothered to change the find the references to Exchange and replace them with Active Directory.
I've lost track of how many times I've reinstalled the UF. Once was because I intuitively chose what I wanted in the custom configuration only to learn later that if you do, all events go to the default index in the troubleshooting section of the instructions.
... View more