Deployment Architecture

Deployment Monitor not getting data to the summary_* indexes

mikelanghorst
Motivator

When I go into the DeploymentMonitor app to All Sourcetypes, the reports show No Results. In fact searching: index=summary_sourcetypes also shows no data. So looking at my search head and indexers, I have no data in $SPLUNK_DB/summary_sourcetypes at all.

When I clicked on the "flush and backfill summary indexes" seems to do little more than creating a very large number of jobs in the dispatch directory on the search head.

What am I missing here? The indexes are created on the indexers and the search head, and other data is forwarded just fine from the search head to the indexer.

1 Solution

mikelanghorst
Motivator

With the assistance of Genti on IRC, we found the issue:

I'd configured the Search Head as a SplunkForwarder, to send the data to my indexers. This wasn't routing the license_usage file to the indexers, indicated by the following in "cmd btool outputs list":

forwardedindex.1.blacklist = _.*

I've added a monitor for that specific file to route it, adding:
[monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log]
_TCP_ROUTING = *
index = _internal

Now my searches are returning data for this source

View solution in original post

mikelanghorst
Motivator

With the assistance of Genti on IRC, we found the issue:

I'd configured the Search Head as a SplunkForwarder, to send the data to my indexers. This wasn't routing the license_usage file to the indexers, indicated by the following in "cmd btool outputs list":

forwardedindex.1.blacklist = _.*

I've added a monitor for that specific file to route it, adding:
[monitor://$SPLUNK_HOME/var/log/splunk/license_usage.log]
_TCP_ROUTING = *
index = _internal

Now my searches are returning data for this source

Genti
Splunk Employee
Splunk Employee

I'm putting my name here just to:

Splunk > Trolling for upgoats!

0 Karma

mikelanghorst
Motivator

Just realized I didn't actually answer your question alex. The sources are enabled.

0 Karma

mikelanghorst
Motivator

Following the trail back from the saved search "All sourcetypes regenerator" the macro sourcetype_metrics didn't work. Which led me to: index=_internal source=*license_usage.log, which also had no data. The tailing processor says the file is being read (100%), but can't currently find the data.

0 Karma

mikelanghorst
Motivator

Hmm, maybe it's just the saved searches aren't scheduled to feed these reports? I figured if there was a default report that the required search would be scheduled by default.

0 Karma

mikelanghorst
Motivator

/app/splunk/var/log/splunk is indeed enabled, and I can search for splunkd messages. But not seeing any messages related to summary_sourcetypes in splunkd.log.

I'm running 4.2.3 with no search head pooling. Looking at each summary_* index:
summary_forwarders - have buckets here
summary_hosts - no buckets
summary_indexers - have buckets here
summary_pools - no buckets
summary_sources - no buckets
summary_sourcetypes - no buckets

araitz
Splunk Employee
Splunk Employee

Can you verify that your issue is not the same as http://splunk-base.splunk.com/answers/34532/deployment-monitor-issue-no-data-in-summary-indexes ?

If not, are you running search head pooling? What version of Splunk are you running?

Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...