I just added a new Universal Forwarder to our Splunk deployment (we previously were running everything on a single server, this is the first attempt at a Forwarder/Receiver). For the most part, everything seems to be working okay. We're receiving data on the indexer, able to search it, etc.
Then I enabled the Deployment Monitor app, but it not showing any data. It seems that our summary* indexes are empty (if I run a search with index=summary or index=summary_indexers, I get no results)
I do see jobs running in the Searches & Reports management interface, and I've also tried to backfill the data inside of Deployment Monitor, with no luck.
I see the following log entries in splunkd.log regarding the summary indexes. This repeats for all of the summary indexes (summary, summaryfowarders, summaryhosts, summarypools, summarysources, summary_sourcetypes).
11-16-2011 16:15:09.484 -0700 INFO IndexProcessor - Initializing index: summary 11-16-2011 16:15:09.484 -0700 INFO HotDBManager - setting hot mgr params: /opt/splunk/var/lib/splunk/summarydb/db maxHotSpanSecs=7776000 maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000 11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - index summary initialized with [300,60,188697600,,,,786432000,20,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615ms] 11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - openDatabase for /opt/splunk/var/lib/splunk/summarydb/db 11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - We are running on a pre-existing database opening ... 11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - No databases found starting fresh ! 11-16-2011 16:15:09.484 -0700 INFO databasePartitionPolicy - CREATION TIME for /opt/splunk/var/lib/splunk/summarydb/db : 1321481049 11-16-2011 16:15:09.484 -0700 WARN databasePartitionPolicy - failed to open metadata for /opt/splunk/var/lib/splunk/summarydb/db, will attempt full rebuild 11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - rebuildMetadata called: full=true path=/opt/splunk/var/lib/splunk/summarydb/db reason=initopenMetaData failed 11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - clearing existing internal aggregate metadata (/opt/splunk/var/lib/splunk/summarydb/db) 11-16-2011 16:15:09.485 -0700 INFO databasePartitionPolicy - currentId for /opt/splunk/var/lib/splunk/summarydb/db after openDatabases = 0
What do you see if you search your indexer's internal index for the following:
index="_internal" source="*metrics.log" group=tcpin_connections"
Specifically, do you see any events from your universal forwarder (i.e. host=yourufhost_name)?
Found the issue. Our system/local/inputs.conf file on our indexer, for some reason, had this:
[monitor://$SPLUNK_HOME/var/log/splunk] disabled = 1
I'm not sure why this was there, probably some relic of the past, but re-enabling this monitor caused everything to start working with the deployment monitor.
That's a bit odd, "disabled = 0" would indicate that the input was, in fact, enabled - as is expected.
Thanks, I edited my answer to read disabled = 1. I initially pasted in my corrected version by accident.
@apringle after 18 months, I LOVE YOU. I had the same problem and i fix it thanks to your auto-answer 😄
Thank you for the suggestions - this search returned nothing, which caused me to dig into this and find the solution.