Splunk for VMware 2.0 is killing my license

[saccam447]

I have a 5 host VMware Cluster in my new datacenter that I am building and it is totally killing my 10GB license. all of the videos say that each host will take 600MB to 1GB in license So i guessing it will take 6-7GB worst case scenario. Right now Im already over 5GB and its not even noon. There are currently only 11 vms running in my cluster that are averaging on the idle side. is there any i can do to turn down the frequency or the volume of data the FA VM is sending over.

P.S. I have no other forwarders sending data over. Also I have syslog configured for the hosts in the cluster that take less than 50 MB/day

Please help

[bbingham]

The vmware engine is very configurable and there are a lot of options for tuning down the collection. The first thing that we need to know is what's really blowing out your license, run the following from a splunk search:

index="_internal" group=per_sourcetype_thruput | stats sum(kb) as Data_In_KB by series | eval Data_In_MB=(Data_In_KB/1024) | sort - Data_In_MB

And post the results. This app is a splunk supported app, so support can work directly with you on your issue once a support ticket is open. I'll do what I can without seeing how you have it configured, but more then likely we may need to have you upload a diag from the FA and the splunk search head.

[jeff]

I'm not sure whether this is the case with the VMWare app, but defined file monitor inputs point to a file directory will consume all of the logs it can find. If this is a new instance of Splunk looking at data, you may be bringing in older data. Once it catches up things would settle down. You could figure out pretty quickly if this is the case by doing a Real Time search over all time- if you're seeing data far in the past (ie, not within seconds of now) this might be a contributing factor to the volume you're seeing.

A couple of things to consider:

1. You can have 4 license violations within a rolling 30 day window without blocking your access to use Splunk (though you'll be presented with a nice little persistent message reminding you of the fact)
2. After the 5th violation, you will still be able to consume data, but will be blocked from searching until the violations are cleared or age out.
3. There is a "ignoreOlderThan" parameter you can specify in inputs.conf (again, if this is the issue), though once Splunk has seen the file for the first time it's too late - the parameter will be ignored. See http://docs.splunk.com/Documentation/Splunk/latest/admin/inputsconf

[pwattssplunk]

Have you looked at the data coming in? If there is a particularly noisy problem in your vmware environment that can account for a large portion of the incoming data. Any estimation of how much data will be generated from a particular system type can only be an estimation of what is considered average, and can't account for particulars in your configuration or unpredicted errors in your environment.

[saccam447]

Yes I have looked at the data that is coming in. I have checked all of the performance metrics, which is CPU, MEM and NET for all of the VMS and hosts which im confident that doesnt not take that much. The only chatty box on my network is the FA VM which is averaging 50%-75% on 4 cpus. The security logs that have come in. VMotion is enabled but I dont move VMs since im still in the early stages of building out the environment. there are only two small datastores connected currently. I will be creating more aggregates later next week but for the sake of this
Q, there isnt much.