Couple of more things I did. Now I collect 500MB a day for whole vmware index, it includes 4 hosts, on vc centre and like 80 vm guests.
This steps will reduce data without lose of existing functionality:
/home/splunkadmin/opt/splunk/etc/apps/Splunk_TA_vmware/local/engineinvvc1.conf
action = InventoryDiscovery
inventoryLevel=Required
interval = 3600
This will further reduce inventory data.
On indexer add nullQuies for some logs events:
apps/Splunk_TA_vcenter/local
props.conf
[host::myindexerhost_changeit]
TZ = America/Toronto
TRANSFORMS-vm = vmnull
TRANSFORMS-vm2 = vmnull-2
[vmware:esxlog:hostd]
TRANSFORMS-vm = vmnull
TRANSFORMS-vm2 = vmnull-2
[vmware:esxlog:vpxa]
TRANSFORMS-vm3 = vmnull-3
[vmware:vclog:vpxd]
TRANSFORMS-vm4 = vmnull-4
in transforms.conf
[vmnull]
REGEX=SSL\sHandshake\sfailed
DEST_KEY=queue
FORMAT=nullQueue
[vmnull-2]
REGEX=SSL_accept\sfailed
DEST_KEY=queue
FORMAT=nullQueue
[vmnull-3]
REGEX=info\s\'Default
DEST_KEY=queue
FORMAT=nullQueue
[vmnull-4]
REGEX=Not\scollecting\sstats\sthis\stime
DEST_KEY=queue
FORMAT=nullQueue
Next steps will reduce amount of performance data you are getting with loss of some functionality.
I have performance monitoring done by different agents, with splunk I want to monitor only VM infrastructure, hosts performance and VM guests disk latency. You can use similar technique to fine tune collecting data. More details at: http://docs.splunk.com/Documentation/VMW/latest/Install/engine.confsettings
Next steps will help to achieve it.
In each engineperf.conf file (you might have multiple) make next changes:
For example Original:
for every stanza with action=PerfDiscovery add next stanza:
perfLevel=2
(you can set level 3 if you notice something missing, difference described here: http://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.HistoricalInterval.html).
perfManagedEntityWhitelist will deactivate collection perfdata from Guest and virtual appliances, but it will collect full perf info from host, datastores, clusters etc.
perfManagedEntityWhitelist = ClusterComputeResource|ResourcePool|HostSystem
Now to enable collection of specific metrix from guests copy stanza you just modify (add something to stanza name, like -disk) and replace
perfManagedEntityWhitelist = ClusterComputeResource|ResourcePool|HostSystem
to VirtualMachine (you can add |VirtualApp if you need) and use perfTypeWhitelist to add perftype you want to collect.
For example this:
[esx4host1]
url = https://host1/sdk/webService
username = splunkforvmuser
password = xxxxxxxxxx
action = PerfDiscovery
perfLevel=2
perfInstanceData = OFF
interval = 60
perfManagedEntityWhitelist = ClusterComputeResource|ResourcePool|HostSystem
will become this:
[esx4host1-disk]
url = https://host1/sdk/webService
username = splunkforvmuser
password = xxxxxxxxxx
action = PerfDiscovery
perfLevel=2
perfInstanceData = OFF
interval = 60
perfManagedEntityWhitelist = VirtualMachine
perfTypeWhitelist=disk
With this you can separate what performance you collect from host and guests. To disk you can add any of the following:
cpu
disk
net
mem
power
ds (datastore)
cl (cluster services)
ma (management agent)
sa (storage adapter)
spth (storage path)
rcpu (resource scheduler)
vdsk (virtual disk)
vcdbg (vc debug info)
vcres (vc resources)
sys (system)
From what I see splunk Big Data approach is to grab all data it can and them figure out what to do with it. This is valid approach if you have unlimited license, if not you are paying price for indexing useless data. We can not afford it, so I had to do all this tweaking to get under our license limit, yet have important data for analysis.
... View more