Getting Data In
Highlighted

Why is cluster master not showing custom indexes in Indexer Clustering:Master Node Dashboard?

SplunkTrust
SplunkTrust

I have a test environment and a production environment and have what appear to be identical settings with different results. In our production environment I am able to see our newly created indexes in the Indexer Clusters:Master Node dashboard but can only see internal and default indexes in the test environment. Using ./splunk cmd btool indexes list --debug on an indexer in each environment I compared the output for one of the indexes that is in both production and test and am unable to see a difference that may cause this. The only difference in values are in the frozenTimePeriodInSecs, maxDataSize, maxHotSpanSecs. and tstatsHomepath values. I have seen it suggested that the issue is with having repFactor not set to auto. For both environments we have repFactor=0 and it appears in one but not the other. I have verified there is data in the indexes for both environments.

Any ideas why the indexes appear in production but not test? Here are the outputs of the btool:

Test:

/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-orgspecific-TEST/local/indexes.conf [test]
/opt/ndx1/etc/system/default/indexes.conf assureUTF8 = false
/opt/ndx1/etc/system/default/indexes.conf bucketRebuildMemoryHint = auto
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-orgspecific-TEST/local/indexes.conf coldPath = $SPLUNKDB/email/colddb
/opt/ndx1/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0
/opt/ndx1/etc/system/default/indexes.conf coldToFrozenDir =
/opt/ndx1/etc/system/default/indexes.conf coldToFrozenScript =
/opt/ndx1/etc/system/default/indexes.conf compressRawdata = true
/opt/ndx1/etc/system/default/indexes.conf defaultDatabase = main
/opt/ndx1/etc/system/default/indexes.conf enableDataIntegrityControl = false
/opt/ndx1/etc/system/default/indexes.conf enableOnlineBucketRepair = true
/opt/ndx1/etc/system/default/indexes.conf enableRealtimeSearch = true
/opt/ndx1/etc/system/default/indexes.conf enableTsidxReduction = false
/opt/ndx1/etc/system/default/indexes.conf frozenTimePeriodInSecs = 188697600
/opt/ndx1/etc/slave-apps/DS-all
departments-IDX-orgspecific-TEST/local/indexes.conf homePath = $SPLUNKDB/email/db
/opt/ndx1/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0
/opt/ndx1/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10
/opt/ndx1/etc/system/default/indexes.conf indexThreads = auto
/opt/ndx1/etc/system/default/indexes.conf journalCompression = gzip
/opt/ndx1/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d
/opt/ndx1/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0
/opt/ndx1/etc/system/default/indexes.conf maxConcurrentOptimizes = 6
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-defaultindexes/default/indexes.conf maxDataSize = 10000
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-defaultindexes/default/indexes.conf maxHotBuckets = 10
/opt/ndx1/etc/system/default/indexes.conf maxHotIdleSecs = 0
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-orgspecific-TEST/local/indexes.conf maxHotSpanSecs = 86400
/opt/ndx1/etc/system/default/indexes.conf maxMemMB = 5
/opt/ndx1/etc/system/default/indexes.conf maxMetaEntries = 1000000
/opt/ndx1/etc/system/default/indexes.conf maxRunningProcessGroups = 8
/opt/ndx1/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1
/opt/ndx1/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300
/opt/ndx1/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-defaultindexes/default/indexes.conf maxTotalDataSizeMB = 5000000
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-defaultindexes/default/indexes.conf maxWarmDBCount = 4000
/opt/ndx1/etc/system/default/indexes.conf memPoolMB = auto
/opt/ndx1/etc/system/default/indexes.conf minHotIdleSecsBeforeForceRoll = auto
/opt/ndx1/etc/system/default/indexes.conf minRawFileSyncSecs = disable
/opt/ndx1/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000
/opt/ndx1/etc/system/default/indexes.conf partialServiceMetaPeriod = 0
/opt/ndx1/etc/system/default/indexes.conf processTrackerServiceInterval = 1
/opt/ndx1/etc/system/default/indexes.conf quarantineFutureSecs = 2592000
/opt/ndx1/etc/system/default/indexes.conf quarantinePastSecs = 77760000
/opt/ndx1/etc/system/default/indexes.conf rawChunkSizeBytes = 131072
/opt/ndx1/etc/system/default/indexes.conf repFactor = 0
/opt/ndx1/etc/system/default/indexes.conf rotatePeriodInSecs = 60
/opt/ndx1/etc/system/default/indexes.conf rtRouterQueueSize = 10000
/opt/ndx1/etc/system/default/indexes.conf rtRouterThreads = 0
/opt/ndx1/etc/system/default/indexes.conf serviceMetaPeriod = 25
/opt/ndx1/etc/system/default/indexes.conf serviceOnlyAsNeeded = true
/opt/ndx1/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30
/opt/ndx1/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
/opt/ndx1/etc/system/default/indexes.conf suppressBannerList =
/opt/ndx1/etc/system/default/indexes.conf suspendHotRollByDeleteQuery = false
/opt/ndx1/etc/system/default/indexes.conf sync = 0
/opt/ndx1/etc/system/default/indexes.conf syncMeta = true
/opt/ndx1/etc/slave-apps/DS-alldepartments-IDX-orgspecific-TEST/local/indexes.conf thawedPath = $SPLUNKDB/email/thaweddb
/opt/ndx1/etc/system/default/indexes.conf throttleCheckPeriod = 15
/opt/ndx1/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800
/opt/ndx1/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600
/opt/ndx1/etc/system/default/indexes.conf tstatsHomePath = volume:
splunksummaries/$indexname/datamodelsummary
/opt/ndx1/etc/system/default/indexes.conf warmToColdScript =

Production:

/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      [test]
/opt/splunk/etc/system/default/indexes.conf                                            assureUTF8 = false
/opt/splunk/etc/system/default/indexes.conf                                            bucketRebuildMemoryHint = auto
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      coldPath = volume:cold/email/colddb
/opt/splunk/etc/system/default/indexes.conf                                            coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf                                            coldToFrozenDir =
/opt/splunk/etc/system/default/indexes.conf                                            coldToFrozenScript =
/opt/splunk/etc/system/default/indexes.conf                                            compressRawdata = true
/opt/splunk/etc/system/default/indexes.conf                                            defaultDatabase = main
/opt/splunk/etc/system/default/indexes.conf                                            enableDataIntegrityControl = false
/opt/splunk/etc/system/default/indexes.conf                                            enableOnlineBucketRepair = true
/opt/splunk/etc/system/default/indexes.conf                                            enableRealtimeSearch = true
/opt/splunk/etc/system/default/indexes.conf                                            enableTsidxReduction = false
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              frozenTimePeriodInSecs = 31536000
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      homePath = volume:hot/email/db
/opt/splunk/etc/system/default/indexes.conf                                            homePath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf                                            hotBucketTimeRefreshInterval = 10
/opt/splunk/etc/system/default/indexes.conf                                            indexThreads = auto
/opt/splunk/etc/system/default/indexes.conf                                            journalCompression = gzip
/opt/splunk/etc/system/default/indexes.conf                                            maxBloomBackfillBucketAge = 30d
/opt/splunk/etc/system/default/indexes.conf                                            maxBucketSizeCacheEntries = 0
/opt/splunk/etc/system/default/indexes.conf                                            maxConcurrentOptimizes = 6
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxDataSize = auto_high_volume
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxHotBuckets = 10
/opt/splunk/etc/system/default/indexes.conf                                            maxHotIdleSecs = 0
/opt/splunk/etc/system/default/indexes.conf                                            maxHotSpanSecs = 7776000
/opt/splunk/etc/system/default/indexes.conf                                            maxMemMB = 5
/opt/splunk/etc/system/default/indexes.conf                                            maxMetaEntries = 1000000
/opt/splunk/etc/system/default/indexes.conf                                            maxRunningProcessGroups = 8
/opt/splunk/etc/system/default/indexes.conf                                            maxRunningProcessGroupsLowPriority = 1
/opt/splunk/etc/system/default/indexes.conf                                            maxTimeUnreplicatedNoAcks = 300
/opt/splunk/etc/system/default/indexes.conf                                            maxTimeUnreplicatedWithAcks = 60
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxTotalDataSizeMB = 50000000
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-base/local/indexes.conf              maxWarmDBCount = 4000
/opt/splunk/etc/system/default/indexes.conf                                            memPoolMB = auto
/opt/splunk/etc/system/default/indexes.conf                                            minHotIdleSecsBeforeForceRoll = auto
/opt/splunk/etc/system/default/indexes.conf                                            minRawFileSyncSecs = disable
/opt/splunk/etc/system/default/indexes.conf                                            minStreamGroupQueueSize = 2000
/opt/splunk/etc/system/default/indexes.conf                                            partialServiceMetaPeriod = 0
/opt/splunk/etc/system/default/indexes.conf                                            processTrackerServiceInterval = 1
/opt/splunk/etc/system/default/indexes.conf                                            quarantineFutureSecs = 2592000
/opt/splunk/etc/system/default/indexes.conf                                            quarantinePastSecs = 77760000
/opt/splunk/etc/system/default/indexes.conf                                            rawChunkSizeBytes = 131072
/opt/splunk/etc/system/default/indexes.conf                                            repFactor = 0
/opt/splunk/etc/system/default/indexes.conf                                            rotatePeriodInSecs = 60
/opt/splunk/etc/system/default/indexes.conf                                            rtRouterQueueSize = 10000
/opt/splunk/etc/system/default/indexes.conf                                            rtRouterThreads = 0
/opt/splunk/etc/system/default/indexes.conf                                            serviceMetaPeriod = 25
/opt/splunk/etc/system/default/indexes.conf                                            serviceOnlyAsNeeded = true
/opt/splunk/etc/system/default/indexes.conf                                            serviceSubtaskTimingPeriod = 30
/opt/splunk/etc/system/default/indexes.conf                                            streamingTargetTsidxSyncPeriodMsec = 5000
/opt/splunk/etc/system/default/indexes.conf                                            suppressBannerList =
/opt/splunk/etc/system/default/indexes.conf                                            suspendHotRollByDeleteQuery = false
/opt/splunk/etc/system/default/indexes.conf                                            sync = 0
/opt/splunk/etc/system/default/indexes.conf                                            syncMeta = true
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-org_specific/local/indexes.conf      thawedPath = $SPLUNK_DB/email/thaweddb
/opt/splunk/etc/system/default/indexes.conf                                            throttleCheckPeriod = 15
/opt/splunk/etc/system/default/indexes.conf                                            timePeriodInSecBeforeTsidxReduction = 604800
/opt/splunk/etc/system/default/indexes.conf                                            tsidxReductionCheckPeriodInSec = 600
/opt/splunk/etc/slave-apps/DS-all_departments-IDX-Volumes/local/indexes.conf           tstatsHomePath = volume:hot/$_index_name/datamodel_summary
/opt/splunk/etc/system/default/indexes.conf                                            warmToColdScript =
0 Karma
Highlighted

Re: Why is cluster master not showing custom indexes in Indexer Clustering:Master Node Dashboard?

Explorer

hmm I was actually advised by a splunk engineer to set it repFactor to auto.. once i did that i was able to see all my custom indexes.

Highlighted

Re: Why is cluster master not showing custom indexes in Indexer Clustering:Master Node Dashboard?

SplunkTrust
SplunkTrust

That's what is getting me. Looking at related issues it seems like it was resolved by setting that value. I am just curious as to why, without that value, it would still appear in one environment and not the other.

0 Karma