Why is cluster master not showing custom indexes in Indexer Clustering:Master Node Dashboard?

[mdsnmss]

I have a test environment and a production environment and have what appear to be identical settings with different results. In our production environment I am able to see our newly created indexes in the Indexer Clusters:Master Node dashboard but can only see internal and default indexes in the test environment. Using ./splunk cmd btool indexes list --debug on an indexer in each environment I compared the output for one of the indexes that is in both production and test and am unable to see a difference that may cause this. The only difference in values are in the frozenTimePeriodInSecs, maxDataSize, maxHotSpanSecs. and tstatsHomepath values. I have seen it suggested that the issue is with having repFactor not set to auto. For both environments we have repFactor=0 and it appears in one but not the other. I have verified there is data in the indexes for both environments.

Any ideas why the indexes appear in production but not test? Here are the outputs of the btool:

Test:

/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-org_specific-TEST/local/indexes.conf [test]
/opt/ndx1/etc/system/default/indexes.conf assureUTF8 = false
/opt/ndx1/etc/system/default/indexes.conf bucketRebuildMemoryHint = auto
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-org_specific-TEST/local/indexes.conf coldPath = $SPLUNK_DB/email/colddb
/opt/ndx1/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0
/opt/ndx1/etc/system/default/indexes.conf coldToFrozenDir =
/opt/ndx1/etc/system/default/indexes.conf coldToFrozenScript =
/opt/ndx1/etc/system/default/indexes.conf compressRawdata = true
/opt/ndx1/etc/system/default/indexes.conf defaultDatabase = main
/opt/ndx1/etc/system/default/indexes.conf enableDataIntegrityControl = false
/opt/ndx1/etc/system/default/indexes.conf enableOnlineBucketRepair = true
/opt/ndx1/etc/system/default/indexes.conf enableRealtimeSearch = true
/opt/ndx1/etc/system/default/indexes.conf enableTsidxReduction = false
/opt/ndx1/etc/system/default/indexes.conf frozenTimePeriodInSecs = 188697600
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-org_specific-TEST/local/indexes.conf homePath = $SPLUNK_DB/email/db
/opt/ndx1/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0
/opt/ndx1/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10
/opt/ndx1/etc/system/default/indexes.conf indexThreads = auto
/opt/ndx1/etc/system/default/indexes.conf journalCompression = gzip
/opt/ndx1/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d
/opt/ndx1/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0
/opt/ndx1/etc/system/default/indexes.conf maxConcurrentOptimizes = 6
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-default_indexes/default/indexes.conf maxDataSize = 10000
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-default_indexes/default/indexes.conf maxHotBuckets = 10
/opt/ndx1/etc/system/default/indexes.conf maxHotIdleSecs = 0
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-org_specific-TEST/local/indexes.conf maxHotSpanSecs = 86400
/opt/ndx1/etc/system/default/indexes.conf maxMemMB = 5
/opt/ndx1/etc/system/default/indexes.conf maxMetaEntries = 1000000
/opt/ndx1/etc/system/default/indexes.conf maxRunningProcessGroups = 8
/opt/ndx1/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1
/opt/ndx1/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300
/opt/ndx1/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-default_indexes/default/indexes.conf maxTotalDataSizeMB = 5000000
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-default_indexes/default/indexes.conf maxWarmDBCount = 4000
/opt/ndx1/etc/system/default/indexes.conf memPoolMB = auto
/opt/ndx1/etc/system/default/indexes.conf minHotIdleSecsBeforeForceRoll = auto
/opt/ndx1/etc/system/default/indexes.conf minRawFileSyncSecs = disable
/opt/ndx1/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000
/opt/ndx1/etc/system/default/indexes.conf partialServiceMetaPeriod = 0
/opt/ndx1/etc/system/default/indexes.conf processTrackerServiceInterval = 1
/opt/ndx1/etc/system/default/indexes.conf quarantineFutureSecs = 2592000
/opt/ndx1/etc/system/default/indexes.conf quarantinePastSecs = 77760000
/opt/ndx1/etc/system/default/indexes.conf rawChunkSizeBytes = 131072
/opt/ndx1/etc/system/default/indexes.conf repFactor = 0
/opt/ndx1/etc/system/default/indexes.conf rotatePeriodInSecs = 60
/opt/ndx1/etc/system/default/indexes.conf rtRouterQueueSize = 10000
/opt/ndx1/etc/system/default/indexes.conf rtRouterThreads = 0
/opt/ndx1/etc/system/default/indexes.conf serviceMetaPeriod = 25
/opt/ndx1/etc/system/default/indexes.conf serviceOnlyAsNeeded = true
/opt/ndx1/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30
/opt/ndx1/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
/opt/ndx1/etc/system/default/indexes.conf suppressBannerList =
/opt/ndx1/etc/system/default/indexes.conf suspendHotRollByDeleteQuery = false
/opt/ndx1/etc/system/default/indexes.conf sync = 0
/opt/ndx1/etc/system/default/indexes.conf syncMeta = true
/opt/ndx1/etc/slave-apps/DS-all_departments-IDX-org_specific-TEST/local/indexes.conf thawedPath = $SPLUNK_DB/email/thaweddb
/opt/ndx1/etc/system/default/indexes.conf throttleCheckPeriod = 15
/opt/ndx1/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800
/opt/ndx1/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600
/opt/ndx1/etc/system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
/opt/ndx1/etc/system/default/indexes.conf warmToColdScript =

Production:

/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      [test]
/opt/splunk/etc/system/default/indexes.conf                                            assureUTF8 = false
/opt/splunk/etc/system/default/indexes.conf                                            bucketRebuildMemoryHint = auto
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      coldPath = volume:cold/email/colddb
/opt/splunk/etc/system/default/indexes.conf                                            coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf                                            coldToFrozenDir =
/opt/splunk/etc/system/default/indexes.conf                                            coldToFrozenScript =
/opt/splunk/etc/system/default/indexes.conf                                            compressRawdata = true
/opt/splunk/etc/system/default/indexes.conf                                            defaultDatabase = main
/opt/splunk/etc/system/default/indexes.conf                                            enableDataIntegrityControl = false
/opt/splunk/etc/system/default/indexes.conf                                            enableOnlineBucketRepair = true
/opt/splunk/etc/system/default/indexes.conf                                            enableRealtimeSearch = true
/opt/splunk/etc/system/default/indexes.conf                                            enableTsidxReduction = false
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              frozenTimePeriodInSecs = 31536000
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      homePath = volume:hot/email/db
/opt/splunk/etc/system/default/indexes.conf                                            homePath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf                                            hotBucketTimeRefreshInterval = 10
/opt/splunk/etc/system/default/indexes.conf                                            indexThreads = auto
/opt/splunk/etc/system/default/indexes.conf                                            journalCompression = gzip
/opt/splunk/etc/system/default/indexes.conf                                            maxBloomBackfillBucketAge = 30d
/opt/splunk/etc/system/default/indexes.conf                                            maxBucketSizeCacheEntries = 0
/opt/splunk/etc/system/default/indexes.conf                                            maxConcurrentOptimizes = 6
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxDataSize = auto_high_volume
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxHotBuckets = 10
/opt/splunk/etc/system/default/indexes.conf                                            maxHotIdleSecs = 0
/opt/splunk/etc/system/default/indexes.conf                                            maxHotSpanSecs = 7776000
/opt/splunk/etc/system/default/indexes.conf                                            maxMemMB = 5
/opt/splunk/etc/system/default/indexes.conf                                            maxMetaEntries = 1000000
/opt/splunk/etc/system/default/indexes.conf                                            maxRunningProcessGroups = 8
/opt/splunk/etc/system/default/indexes.conf                                            maxRunningProcessGroupsLowPriority = 1
/opt/splunk/etc/system/default/indexes.conf                                            maxTimeUnreplicatedNoAcks = 300
/opt/splunk/etc/system/default/indexes.conf                                            maxTimeUnreplicatedWithAcks = 60
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxTotalDataSizeMB = 50000000
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxWarmDBCount = 4000
/opt/splunk/etc/system/default/indexes.conf                                            memPoolMB = auto
/opt/splunk/etc/system/default/indexes.conf                                            minHotIdleSecsBeforeForceRoll = auto
/opt/splunk/etc/system/default/indexes.conf                                            minRawFileSyncSecs = disable
/opt/splunk/etc/system/default/indexes.conf                                            minStreamGroupQueueSize = 2000
/opt/splunk/etc/system/default/indexes.conf                                            partialServiceMetaPeriod = 0
/opt/splunk/etc/system/default/indexes.conf                                            processTrackerServiceInterval = 1
/opt/splunk/etc/system/default/indexes.conf                                            quarantineFutureSecs = 2592000
/opt/splunk/etc/system/default/indexes.conf                                            quarantinePastSecs = 77760000
/opt/splunk/etc/system/default/indexes.conf                                            rawChunkSizeBytes = 131072
/opt/splunk/etc/system/default/indexes.conf                                            repFactor = 0
/opt/splunk/etc/system/default/indexes.conf                                            rotatePeriodInSecs = 60
/opt/splunk/etc/system/default/indexes.conf                                            rtRouterQueueSize = 10000
/opt/splunk/etc/system/default/indexes.conf                                            rtRouterThreads = 0
/opt/splunk/etc/system/default/indexes.conf                                            serviceMetaPeriod = 25
/opt/splunk/etc/system/default/indexes.conf                                            serviceOnlyAsNeeded = true
/opt/splunk/etc/system/default/indexes.conf                                            serviceSubtaskTimingPeriod = 30
/opt/splunk/etc/system/default/indexes.conf                                            streamingTargetTsidxSyncPeriodMsec = 5000
/opt/splunk/etc/system/default/indexes.conf                                            suppressBannerList =
/opt/splunk/etc/system/default/indexes.conf                                            suspendHotRollByDeleteQuery = false
/opt/splunk/etc/system/default/indexes.conf                                            sync = 0
/opt/splunk/etc/system/default/indexes.conf                                            syncMeta = true
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      thawedPath = $SPLUNK_DB/email/thaweddb
/opt/splunk/etc/system/default/indexes.conf                                            throttleCheckPeriod = 15
/opt/splunk/etc/system/default/indexes.conf                                            timePeriodInSecBeforeTsidxReduction = 604800
/opt/splunk/etc/system/default/indexes.conf                                            tsidxReductionCheckPeriodInSec = 600
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-Volumes/local/indexes.conf           tstatsHomePath = volume:hot/$_index_name/datamodel_summary
/opt/splunk/etc/system/default/indexes.conf                                            warmToColdScript =

[sudoiamroot]

hmm I was actually advised by a splunk engineer to set it repFactor to auto.. once i did that i was able to see all my custom indexes.

[mdsnmss]

That's what is getting me. Looking at related issues it seems like it was resolved by setting that value. I am just curious as to why, without that value, it would still appear in one environment and not the other.

[kundanshekhx]

Faced exactly the same issue. Issue resolved after adding "repFactor = auto" in test though in production everything is working fine without "repFactor = auto".  

[isoutamo]

Hi

I’m not sure if this is valid for your environment, but there is bug for showing new indexes without restart before 8.0.2. It has occurred in one of our managed environment before we update it to use volumes. After that it disappears.

You seems to use FS on test and volume on prod. And as other already said you must use repFactor = auto with clustering. 

r. Ismo