It means the search for the data did not return the results. This can happen if in a particular run of the KPI there just wasn't data for some of those entities. To debug you should check for a prior run of that KPI/run the generated search from the configuration page's KPI editing modal. You want to see if the entities are represented in those search results. You want to ensure that the data is present that would be mapped to those KPIs and that it arrives in time. One common root cause is late arriving data, in the final step of the KPI there is a section for the monitoring lag and it has a check recommended lag link. Click that link to check for late arriving data. ... View more

You should never have been experiencing connections from your indexers to your DCN's. That would indicate a configuration file called ta_vmware_collection.conf and a file named hydra_node.conf existed in the local directory of Splunk_TA_vmware package installed on your indexers. That also means there is an inputs.conf on your indexers in your Splunk_TA_vmware with ta_vmware_collection_scheduler turned on. TA vmare's collection scheduler modular input should only be run from one machine, typically the search head. If you are in a search head pooled environment you would have had to move the scheduler process off the pool to avoid duplicated work but not onto to the indexers. You normally just use one of the DCN's. The best course for remediation is to simply remove the inputs, ta_vmware_collection, and hydra_node conf files from all your indexers. Your problems will then go away as soon as configuration is refreshed or splunk is restarted. ... View more

The "proper" thing to do is to make a data model and then run the aggregations on the fly for your searches in tstats. Accelerate the data model and hopefully it will not be too bad. The other solution is actually to use summary indexing, but to make all the fields you are interested in indexed fields with props and transforms. By making them indexed fields you can still use tstats against the index. In your search query, instead of from datamodel=blah you'd do from index=blah. This still comes with all the problems that tscollect has (data lag causing incorrect data namely), but on the plus side you get the retention policies of indexes and if summary indexing is configured properly it will go out and store the data on your indexers. ,The best you could probably do is to actually do those aggregations on the accelerated data model and not expect them to already be done. That is set up your data model to have the fields and run a tstats on the accelerated model by search_id. It is not pretty, but what you want is only possible in data model acceleration if splunk allowed base search data models to be accelerated which it does not. The other thing you could do is migrate to summary indexing. You can then make all of the fields you care about in your summary index, indexed fields. The result of this is equivalent to making your own tsidx namespace. You can actually still run tstats against that index. Instead of from datamodel=blah, you do from index=blah. The storage technology is the same and it should work with almost the same performance. However the same problems you would have with tscollect (such as data lag causing incorrect data) still exist, whereas they will not in the accelerated data model with aggregations on the fly. ... View more

First a warning and a disclaimer: the supported OS matrix for the data collection node is representative of what was tested in house for both stability and scalability. If you deploy on windows and call support they will likely ask you to redeploy on CentOS or RHEL in order to take the support case further. However, the system was designed to work on any OS and runs with the python distribution that ships with splunk. This means that it is very likely that if you were to deploy on windows everything would work. If there is some issue post back as I would like to get it into the pipeline for a patch or a future maintenance release if not provide you with some ad hoc patch to get over the issue. To check for issues when you have completed the install perform the following search: index=_internal source=*hydra* error ... View more

Yes there is a way to install a single box installation. But first, As a disclaimer, the resources involved in collecting data often make it a poor choice to colocate with a search head or indexer. There are no sizing guidelines for a single box installation and if call support for any issues the first thing you will be asked to do is to move data collection to another machine. Now that that is over on to the how to do it. The Splunk App for NetApp Data ONTAP utilizes the Hydra data collection framwork (that's that SA-Hydra app that ships with it). It follows a fairly ordinary client server paradigm if you consider the scheduler node (typically run at the search head) the client and the data collection nodes as servers with data collection services. So to install on a single box you need simply make that box both the client and the server. This amounts to a normal install except that when you come to the add data collection nodes step you simply add the splunk instance you are configuring the scheduler on as its own data collection node. Thus your data collection node's management URI would be along the lines of https://localhost:8089. Note there is also a security concern with this methodology as this machine will have its own service account stored for logins. With that and the resourcing concern I would really only do this for as simple a POC as possible and never in production. ... View more

JSChart is reading the fields in your search results to do this. I am guessing that you are not using timechart to plot this data which is why that metadata field it is looking for is absent. You can hack it in by ending your search with an eval or you can use a timechart command with a span declaration. for the hack, the integer represents the jschart's x-axis span: | eval _span=60 ... View more

The FA VM is really just a centos VM. Following the instructions for the upgrade of vmware tools on centos machines is all you need. It should be installed in the default installation directory. ... View more

I think you may have to leave the world of simple and form xml behind and move into full blown view (advanced) xml. This will do the trick in advanced xml (assuming I made no errors I do not have your searches and thus was unable to test): ManagedServiceRuns 1 * False 1 splunk.search.job True dashboard client client client clientlistForJobs False thisclient_setting thisclient_setting $target$ True stringreplace False Search True True True sourcetype=jobinfo client="$thisclient$" | fields averageReportsSize |timechart max(averageReportsSize) ManagedServiceMaxTime Chart charting.chart.nullValueMode charting.chart charting.legend.showMarkers True False True connect line true 100% flashtimeline flashtimeline ManagedServiceRuns Positions Attempted Chart charting.chart.nullValueMode charting.chart charting.legend.showMarkers True False True connect line true 100% flashtimeline flashtimeline ManagedServiceRuns Averrage ReportSizeChart charting.chart.nullValueMode charting.chart charting.legend.showMarkers True False True connect line true 100% flashtimeline flashtimeline ... View more