Splunk App for VMware: Why do I not see performance data?

[dominiquevocat]

Update: I narrowed the issue down to the searches in SA-VMW-PERFORMANCE, the scheduled searches that fill the tsidx's fail.

The searches look like this:
sourcetype=vmware:perf:mem source=VMPerf:VirtualMachine [stats count | addinfo | eval cutoff=info_max_time-300 | eval prefix="indextime>" | eval search=prefix+cutoff | table search] | lookup FullHierarchy host, moid OUTPUT parent as hs, rp | lookup FullHierarchy host, moid as hs OUTPUT parent, parentType | eval ccr=if(parentType=="ClusterComputeResource", parent, "N/A") | table _time p* instance moid host hs rp ccr | tscollect(vmw_perf_mem_virtualmachine)

And don't run. Slightly modifying it so its syntactically working (example below) generates the tsidx but - seriously?
sourcetype=vmware:perf:mem source=VMPerf:VirtualMachine | append [stats count | addinfo | eval cutoff=info_max_time-300 | eval prefix="indextime>" | eval search=prefix+cutoff | table search] | lookup FullHierarchy host, moid OUTPUT parent as hs, rp | lookup FullHierarchy host, moid as hs OUTPUT parent, parentType | eval ccr=if(parentType=="ClusterComputeResource", parent, "N/A") | table _time p* instance moid host hs rp ccr| tscollect namespace=vmw_perf_mem_virtualmachine squashcase=true keepresults=true

---

I deployed the various components for the Splunk App for VMware (dcn, vcenter, indexer apps, searchhead apps) and for all I can see, there is the relevant data. However the performance data is not visible.

When I run the search d_home_proactive_monitoring_row1_col1_gauge3, I get:

x   y1     y2      y3
0   0.500   0.750   1

I think i have all sorts of data available. The searches populating the various performance indicators seem to work up to the last step where there is a macro applied to the data.

In the Proactive Monitoring view I get:
"No results from performance search, please check that you have performance data and that your tsidx namespaces are being populated with it."

sourcetype  count(sourcetype)
vmware:esxlog:Fdm   13
vmware:esxlog:Hostd 416
vmware:esxlog:Vpxa  1251
vmware:esxlog:crond 30
vmware:esxlog:hostd-probe   711
vmware:esxlog:sfcb-vmware   18
vmware:esxlog:snmpd 360
vmware:esxlog:syslog    63
vmware:esxlog:vmkernel  600
vmware:events   60
vmware:inv:clustercomputeresource   6
vmware:inv:datastore    4
vmware:inv:resourcepool 16
vmware:inv:vm   47
vmware:perf:cpu 3124
vmware:perf:datastore   3124
vmware:perf:disk    3124
vmware:perf:hbr 525
vmware:perf:mem 3124
vmware:perf:net 2948
vmware:perf:power   3124
vmware:perf:rescpu  3124
vmware:perf:storageAdapter  525
vmware:perf:storagePath 525
vmware:perf:sys 3124
vmware:perf:virtualDisk 2599
vmware:vclog    182
vmware:vclog:cim-diag   26
vmware:vclog:tomcat 218
vmware:vclog:vpxd   3166
vmware:vclog:vpxd-profiler  56578
vmware:vclog:vws    57

I am puzzled as to what is the issue...
I think I followed the instructions plus the video (and the docs are not very imho).

[Masa]

Sorry for difficulties in deploying the app.

Any chance you filed a Support case so that Splunk Support can take a look into your deployment, logs and configurations?

Otherwise, from the information you provided above;
1. Data are indexed. This means DCN is working
2. tsidx files are not populated by scheduled searches. Assuming all deployment is correct, most common cause of the issue is either;

a) system clock among indexers, search head, DCN and vCenter are off,
b) vmware_admin role is not assigned to a user called admin.

By the way, the scheduled search string you located is a kind of hack to identify indextime range in order to avoid missing events by latency of indexing events.

[dominiquevocat]

Hi, i recopied the SA-VMW-Performance and restared splunk and manuall ran each tscollect search and most were filled then and from that moment on keep being updated! Strange. However the ones listed below do not get filled. At least some of them due to information lacking thus no output i suppse. I will check the teh questions you posted. Thanks

vmw_perf_mem_resourcepool
vmw_perf_cpu_resourcepool
vmw_perf_storagepath_hostsystem
vmw_perf_storageadapter_hostsystem
vmw_perf_hbr_hostsystem
vmw_perf_datastore_hostsystem
vmw_perf_resstorageadapter_hostsystem
vmw_perf_resdisk_virtualmachine
vmw_perf_mem_clustercomputeresource
vmw_perf_cpu_clustercomputeresource
vmw_perf_vmop_clustercomputeresource

[kserra_splunk]

What version of splunk are you running, what version of the vmware app are you running and what operating system is everything running on?

[dominiquevocat]

Splunk 6.2.4
VMware App 3.1.4
The VWware Components are up to date too.
Our Splunk runs on SLES 11 (not Windows, there were some issues with VMware App on Windows, no?)