Internal indexes _audit _internal _introspection and _thefishbucket all showing as disabled

[larryggibson]

On one of our indexers _audit _internal _introspection and _thefishbucket indexes have been marked as disabled as shown in settings-> indexes.

We have checked that there are no duplicate buckets and when we look in splunkd.log we see that the indexes are getting updates without errors.

We only noticed because we were having issues with performance on this indexer and went to look at management console which was not updating completely due to _introspection being disabled. Does anyone have an ideas how we can get these re-enabled or recreated?

[harsmarvania57]

Hi,

Please check indexes.conf settings on that Indexer using btool, please use below command

$SPLUNK_HOME/bin/splunk cmd btool indexes list --debug INDEXNAME

This is most likely due to someone overwrote Indexes config on that Indexer, if indexes setting overwrote locally then you can remove that settings and restart splunk on that Indexer.

[larryggibson]

I checked /opt/splunk/etc/system/local/indexes.conf and it only had an entry in there for the _introspection index. I took a copy of the file and rm ed the local/indexes.conf and restarted splunk. Running btool again I checked the output with a known good output from another indexer which is working fine for the _introspection index and they are identical but the index remains disabled.

[harsmarvania57]

Hi,

Can you please try below command and check whether that index is disabled in any other apps?

/opt/splunk/bin/splunk cmd btool indexes list --debug _introspection | grep disabled

[larryggibson]

That didn't find anything. I checked _audit etc too. It's a weird one. Its only these 4 indexes.

[punyanit]

Hi,

this might be due to Splunk detected a bucket collision :
2 or more of the buckets folders have the same unique ID.

[larryggibson]

I checked the db folders and cant see any duplicate IDs and there is nothing in the splunkd.log.

[larryggibson]

| dbinspect index=_introspection lists the buckets also but not sure whats going on...

Thanks for trying to help harsmarvania57!

[larryggibson]

Rechecked splunkd.log and it all looks good and can see it is updating buckets

08-20-2019 14:24:02.400 +0100 INFO IndexWriter - idx=_introspection, Initializing, params='[300,period=60,frozenTimePeriodInSecs=1209600,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1073741824,optimizeEvery=5,syncMeta=true,maxTotalDataSizeMB=500000,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000,maxMetadataEntries=1000000,maxHotIdleSecs=0,maxHotBuckets=3,quarantinePastSecs=77760000,quarantineFutureSecs=2592000,maxSliceSize=131072,serviceMetaPeriod=25,partialServiceMetaPeriod=0,throttleCheckPeriod=15,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551615,maxBloomBackfillBucketAge_secs=2592000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,tsidxReductionCheckPeriodInSec=600,timePeriodInSecBeforeTsidxReduction=604800]' isSlave=false
08-20-2019 14:24:02.401 +0100 INFO IndexWriter - openDatabases complete currentId=499 idx=_introspection
08-20-2019 14:24:03.311 +0100 INFO IndexWriter - idx=_introspection Creating hot bucket=hot_v1_499, given event timestamped=1566307442
08-20-2019 14:24:03.311 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_introspection~499~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A'
08-20-2019 14:24:03.415 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-20-2019 14:24:04.042 +0100 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~498~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A
08-20-2019 14:24:04.347 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'

No visible errors being called out.

[larryggibson]

Running the debug command gave the following output does look like the symptoms you though?

splunk@spr-splunk-idx01 /opt/splunk/bin> ./splunk cmd btool indexes list --debug _introspection
/opt/splunk/etc/system/local/indexes.conf   [_introspection]
/opt/splunk/etc/system/default/indexes.conf assureUTF8 = false
/opt/splunk/etc/system/local/indexes.conf   bucketRebuildMemoryHint = 0
/opt/splunk/etc/system/default/indexes.conf coldPath = $SPLUNK_DB/_introspection/colddb
/opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf coldToFrozenDir = 
/opt/splunk/etc/system/default/indexes.conf coldToFrozenScript = 
/opt/splunk/etc/system/local/indexes.conf   compressRawdata = 1
/opt/splunk/etc/system/default/indexes.conf defaultDatabase = main
/opt/splunk/etc/system/local/indexes.conf   enableDataIntegrityControl = 0
/opt/splunk/etc/system/local/indexes.conf   enableOnlineBucketRepair = 1
/opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true
/opt/splunk/etc/system/local/indexes.conf   enableTsidxReduction = 0
/opt/splunk/etc/system/local/indexes.conf   frozenTimePeriodInSecs = 1209600
/opt/splunk/etc/system/default/indexes.conf homePath = $SPLUNK_DB/_introspection/db
/opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10
/opt/splunk/etc/system/default/indexes.conf indexThreads = auto
/opt/splunk/etc/system/default/indexes.conf journalCompression = gzip
/opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d
/opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0
/opt/splunk/etc/system/default/indexes.conf maxConcurrentOptimizes = 6
/opt/splunk/etc/system/default/indexes.conf maxDataSize = 1024
/opt/splunk/etc/system/default/indexes.conf maxHotBuckets = 3
/opt/splunk/etc/system/default/indexes.conf maxHotIdleSecs = 0
/opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 7776000
/opt/splunk/etc/system/default/indexes.conf maxMemMB = 5
/opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60
/opt/splunk/etc/system/default/indexes.conf maxTotalDataSizeMB = 500000
/opt/splunk/etc/system/default/indexes.conf maxWarmDBCount = 300
/opt/splunk/etc/system/default/indexes.conf memPoolMB = auto
/opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable
/opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000
/opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0
/opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1
/opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000
/opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000
/opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072
/opt/splunk/etc/system/default/indexes.conf repFactor = 0
/opt/splunk/etc/system/default/indexes.conf rotatePeriodInSecs = 60
/opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25
/opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true
/opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30
/opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
/opt/splunk/etc/system/default/indexes.conf suppressBannerList = 
/opt/splunk/etc/system/default/indexes.conf sync = 0
/opt/splunk/etc/system/local/indexes.conf   syncMeta = 1
/opt/splunk/etc/system/default/indexes.conf thawedPath = $SPLUNK_DB/_introspection/thaweddb
/opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15
/opt/splunk/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800
/opt/splunk/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600
/opt/splunk/etc/system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
/opt/splunk/etc/system/default/indexes.conf warmToColdScript =

[harsmarvania57]

Everything looks good, any error in splunkd.log on Indexer ? If not then I'll suggest you to raise case with Splunk support.