Deployment Architecture

Internal indexes _audit _internal _introspection and _thefishbucket all showing as disabled

larryggibson
Explorer

On one of our indexers _audit _internal _introspection and _thefishbucket indexes have been marked as disabled as shown in settings-> indexes.

We have checked that there are no duplicate buckets and when we look in splunkd.log we see that the indexes are getting updates without errors.

We only noticed because we were having issues with performance on this indexer and went to look at management console which was not updating completely due to _introspection being disabled. Does anyone have an ideas how we can get these re-enabled or recreated?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Please check indexes.conf settings on that Indexer using btool, please use below command

$SPLUNK_HOME/bin/splunk cmd btool indexes list --debug INDEXNAME

This is most likely due to someone overwrote Indexes config on that Indexer, if indexes setting overwrote locally then you can remove that settings and restart splunk on that Indexer.

larryggibson
Explorer

I checked /opt/splunk/etc/system/local/indexes.conf and it only had an entry in there for the _introspection index. I took a copy of the file and rm ed the local/indexes.conf and restarted splunk. Running btool again I checked the output with a known good output from another indexer which is working fine for the _introspection index and they are identical but the index remains disabled.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Can you please try below command and check whether that index is disabled in any other apps?

/opt/splunk/bin/splunk cmd btool indexes list --debug _introspection | grep disabled

larryggibson
Explorer

That didn't find anything. I checked _audit etc too. It's a weird one. Its only these 4 indexes.

0 Karma

punyanit
Path Finder

Hi,

this might be due to Splunk detected a bucket collision :
2 or more of the buckets folders have the same unique ID.

0 Karma

larryggibson
Explorer

I checked the db folders and cant see any duplicate IDs and there is nothing in the splunkd.log.

0 Karma

larryggibson
Explorer

| dbinspect index=_introspection lists the buckets also but not sure whats going on...

Thanks for trying to help harsmarvania57!

0 Karma

larryggibson
Explorer

Rechecked splunkd.log and it all looks good and can see it is updating buckets

08-20-2019 14:24:02.400 +0100 INFO IndexWriter - idx=_introspection, Initializing, params='[300,period=60,frozenTimePeriodInSecs=1209600,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1073741824,optimizeEvery=5,syncMeta=true,maxTotalDataSizeMB=500000,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000,maxMetadataEntries=1000000,maxHotIdleSecs=0,maxHotBuckets=3,quarantinePastSecs=77760000,quarantineFutureSecs=2592000,maxSliceSize=131072,serviceMetaPeriod=25,partialServiceMetaPeriod=0,throttleCheckPeriod=15,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551615,maxBloomBackfillBucketAge_secs=2592000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,tsidxReductionCheckPeriodInSec=600,timePeriodInSecBeforeTsidxReduction=604800]' isSlave=false
08-20-2019 14:24:02.401 +0100 INFO IndexWriter - openDatabases complete currentId=499 idx=_introspection
08-20-2019 14:24:03.311 +0100 INFO IndexWriter - idx=_introspection Creating hot bucket=hot_v1_499, given event timestamped=1566307442
08-20-2019 14:24:03.311 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_introspection~499~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A'
08-20-2019 14:24:03.415 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
08-20-2019 14:24:04.042 +0100 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~498~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A
08-20-2019 14:24:04.347 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'

No visible errors being called out.

0 Karma

larryggibson
Explorer

Running the debug command gave the following output does look like the symptoms you though?

splunk@spr-splunk-idx01 /opt/splunk/bin> ./splunk cmd btool indexes list --debug _introspection
/opt/splunk/etc/system/local/indexes.conf   [_introspection]
/opt/splunk/etc/system/default/indexes.conf assureUTF8 = false
/opt/splunk/etc/system/local/indexes.conf   bucketRebuildMemoryHint = 0
/opt/splunk/etc/system/default/indexes.conf coldPath = $SPLUNK_DB/_introspection/colddb
/opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf coldToFrozenDir = 
/opt/splunk/etc/system/default/indexes.conf coldToFrozenScript = 
/opt/splunk/etc/system/local/indexes.conf   compressRawdata = 1
/opt/splunk/etc/system/default/indexes.conf defaultDatabase = main
/opt/splunk/etc/system/local/indexes.conf   enableDataIntegrityControl = 0
/opt/splunk/etc/system/local/indexes.conf   enableOnlineBucketRepair = 1
/opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true
/opt/splunk/etc/system/local/indexes.conf   enableTsidxReduction = 0
/opt/splunk/etc/system/local/indexes.conf   frozenTimePeriodInSecs = 1209600
/opt/splunk/etc/system/default/indexes.conf homePath = $SPLUNK_DB/_introspection/db
/opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0
/opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10
/opt/splunk/etc/system/default/indexes.conf indexThreads = auto
/opt/splunk/etc/system/default/indexes.conf journalCompression = gzip
/opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d
/opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0
/opt/splunk/etc/system/default/indexes.conf maxConcurrentOptimizes = 6
/opt/splunk/etc/system/default/indexes.conf maxDataSize = 1024
/opt/splunk/etc/system/default/indexes.conf maxHotBuckets = 3
/opt/splunk/etc/system/default/indexes.conf maxHotIdleSecs = 0
/opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 7776000
/opt/splunk/etc/system/default/indexes.conf maxMemMB = 5
/opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8
/opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300
/opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60
/opt/splunk/etc/system/default/indexes.conf maxTotalDataSizeMB = 500000
/opt/splunk/etc/system/default/indexes.conf maxWarmDBCount = 300
/opt/splunk/etc/system/default/indexes.conf memPoolMB = auto
/opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable
/opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000
/opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0
/opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1
/opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000
/opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000
/opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072
/opt/splunk/etc/system/default/indexes.conf repFactor = 0
/opt/splunk/etc/system/default/indexes.conf rotatePeriodInSecs = 60
/opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25
/opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true
/opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30
/opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000
/opt/splunk/etc/system/default/indexes.conf suppressBannerList = 
/opt/splunk/etc/system/default/indexes.conf sync = 0
/opt/splunk/etc/system/local/indexes.conf   syncMeta = 1
/opt/splunk/etc/system/default/indexes.conf thawedPath = $SPLUNK_DB/_introspection/thaweddb
/opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15
/opt/splunk/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800
/opt/splunk/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600
/opt/splunk/etc/system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary
/opt/splunk/etc/system/default/indexes.conf warmToColdScript =
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Everything looks good, any error in splunkd.log on Indexer ? If not then I'll suggest you to raise case with Splunk support.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...