Installed Splunk on Friday, added my AD controllers and my Exchange server, followed all the instructions, and we've done like 60 GB a day for the past few days. Is this just loading in old data or something or is there a switch somewhere I messed up? We have 90 users and a simple domain structure, I can't imagine we're ingesting this much data a day in AD logs.
Setup is very simple: master head with 2 indexers, universal forwarder on each of the Windows hosts, deployment server app setting each universal forwarder to get a copy of Splunk_TA_windows, TA-DNSServer-NT6, and TA-DomainController-2012R2 (we're pure 2012 R2). We also push the indexer IPs via a deployment app. The outputs.conf has both indexers in a single server stanza, which I believe means it load balances?
Either way, I can't justify buying 100GB of license for 6 servers.
I disabled all perfmon on all my Windows hosts, and when I check the indexing volume it tells me it's all one of my AD servers and one of my Exchange servers. Yesterday's total use was 189GB on the windows index, but the entire size of all my indexes is just 20GB across both index servers. I don't understand how log data could be 189GB on just 6 servers.
You can use this view to see which sources are using up a large amount of volume on your Splunk server.
If you are sending perfmon stats from each host you may want to change the interval the metrics are sent at. I've got my boxes all sending stats every 20 seconds (as seen below) and that works just fine. I believe the default for the Windows App is 10 seconds. Below is an example from inputs.conf
counters = % Processor Time;% User Time
instances = _Total
interval = 20
object = Processor