topic Re: Deployment Monitor Issue - no data in summary indexes in Monitoring Splunk

Deployment Monitor Issue - no data in summary indexes

apringle — Mon, 28 Sep 2020 10:07:08 GMT

I just added a new Universal Forwarder to our Splunk deployment (we previously were running everything on a single server, this is the first attempt at a Forwarder/Receiver). For the most part, everything seems to be working okay. We're receiving data on the indexer, able to search it, etc.

Then I enabled the Deployment Monitor app, but it not showing any data. It seems that our summary* indexes are empty (if I run a search with index=summary or index=summary_indexers, I get no results)

I do see jobs running in the Searches & Reports management interface, and I've also tried to backfill the data inside of Deployment Monitor, with no luck.

I see the following log entries in splunkd.log regarding the summary indexes. This repeats for all of the summary indexes (summary, summary_fowarders, summary_hosts, summary_pools, summary_sources, summary_sourcetypes).

11-16-2011 16:15:09.484 -0700 INFO  IndexProcessor - Initializing index: summary
11-16-2011 16:15:09.484 -0700 INFO  HotDBManager - setting hot mgr params: /opt/splunk/var/lib/splunk/summarydb/db maxHotSpanSecs=7776000 maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - index summary initialized with [300,60,188697600,,,,786432000,20,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615ms]
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - openDatabase for /opt/splunk/var/lib/splunk/summarydb/db
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - We are running on a pre-existing database opening ...
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - No databases found starting fresh !
11-16-2011 16:15:09.484 -0700 INFO  databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/summarydb/db : 1321481049
11-16-2011 16:15:09.484 -0700 WARN  databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/summarydb/db, will attempt full rebuild
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/summarydb/db reason=initopenMetaData failed
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - clearing existing internal aggregate metadata (/opt/splunk/var/lib/splunk/summarydb/db)
11-16-2011 16:15:09.485 -0700 INFO  databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/summarydb/db after openDatabases = 0

Re: Deployment Monitor Issue - no data in summary indexes

araitz — Mon, 28 Sep 2020 10:07:26 GMT

What do you see if you search your indexer's internal index for the following:

 index="_internal" source="*metrics.log" group=tcpin_connections"

Specifically, do you see any events from your universal forwarder (i.e. host=your_uf_host_name)?

Re: Deployment Monitor Issue - no data in summary indexes

apringle — Fri, 18 Nov 2011 16:36:47 GMT

Found the issue. Our system/local/inputs.conf file on our indexer, for some reason, had this:

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 1

I'm not sure why this was there, probably some relic of the past, but re-enabling this monitor caused everything to start working with the deployment monitor.

Re: Deployment Monitor Issue - no data in summary indexes

apringle — Fri, 18 Nov 2011 16:37:54 GMT

Thank you for the suggestions - this search returned nothing, which caused me to dig into this and find the solution.

Re: Deployment Monitor Issue - no data in summary indexes

araitz — Fri, 18 Nov 2011 16:39:41 GMT

Very weird! Glad you were able to find the solution.

Re: Deployment Monitor Issue - no data in summary indexes

hexx — Fri, 18 Nov 2011 16:54:40 GMT

That's a bit odd, "disabled = 0" would indicate that the input was, in fact, enabled - as is expected.

Re: Deployment Monitor Issue - no data in summary indexes

apringle — Fri, 18 Nov 2011 16:58:12 GMT

Thanks, I edited my answer to read disabled = 1. I initially pasted in my corrected version by accident.

Re: Deployment Monitor Issue - no data in summary indexes

RiccardoV — Wed, 22 Jan 2014 11:15:25 GMT

@apringle after 18 months, I LOVE YOU. I had the same problem and i fix it thanks to your auto-answer 😄