Deployment Monitor not getting data to the summary_* indexes

[mikelanghorst]

When I go into the DeploymentMonitor app to All Sourcetypes, the reports show No Results. In fact searching: index=summary_sourcetypes also shows no data. So looking at my search head and indexers, I have no data in $SPLUNK_DB/summary_sourcetypes at all.

When I clicked on the "flush and backfill summary indexes" seems to do little more than creating a very large number of jobs in the dispatch directory on the search head.

What am I missing here? The indexes are created on the indexers and the search head, and other data is forwarded just fine from the search head to the indexer.

[mikelanghorst · Accepted solution]

With the assistance of Genti on IRC, we found the issue:

I'd configured the Search Head as a SplunkForwarder, to send the data to my indexers. This wasn't routing the license_usage file to the indexers, indicated by the following in "cmd btool outputs list":

forwardedindex.1.blacklist = _.*

I've added a monitor for that specific file to route it, adding:
[monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log]
_TCP_ROUTING = *
index = _internal

Now my searches are returning data for this source

[Genti]

I'm putting my name here just to:

Splunk > Trolling for upgoats!

[mikelanghorst]

Just realized I didn't actually answer your question alex. The sources are enabled.

[mikelanghorst]

Following the trail back from the saved search "All sourcetypes regenerator" the macro sourcetype_metrics didn't work. Which led me to: index=_internal source=*license_usage.log, which also had no data. The tailing processor says the file is being read (100%), but can't currently find the data.

[mikelanghorst]

Hmm, maybe it's just the saved searches aren't scheduled to feed these reports? I figured if there was a default report that the required search would be scheduled by default.

[mikelanghorst]

/app/splunk/var/log/splunk is indeed enabled, and I can search for splunkd messages. But not seeing any messages related to summary_sourcetypes in splunkd.log.

I'm running 4.2.3 with no search head pooling. Looking at each summary_* index:
summary_forwarders - have buckets here
summary_hosts - no buckets
summary_indexers - have buckets here
summary_pools - no buckets
summary_sources - no buckets
summary_sourcetypes - no buckets

[araitz]

Can you verify that your issue is not the same as http://splunk-base.splunk.com/answers/34532/deployment-monitor-issue-no-data-in-summary-indexes ?

If not, are you running search head pooling? What version of Splunk are you running?