I checked the db folders and cant see any duplicate IDs and there is nothing in the splunkd.log. ... View more

| dbinspect index=_introspection lists the buckets also but not sure whats going on... Thanks for trying to help harsmarvania57! ... View more

Rechecked splunkd.log and it all looks good and can see it is updating buckets 08-20-2019 14:24:02.400 +0100 INFO IndexWriter - idx=_introspection, Initializing, params='[300,period=60,frozenTimePeriodInSecs=1209600,coldToFrozenScript=,coldToFrozenDir=,warmToColdScript=,maxHotBucketSize=1073741824,optimizeEvery=5,syncMeta=true,maxTotalDataSizeMB=500000,maxMemoryAllocationPerHotSliceMB=5,addressCompressBits=5,isReadOnly=false,maxMergizzles=6,maxHotSpanSecs=7776000,maxMetadataEntries=1000000,maxHotIdleSecs=0,maxHotBuckets=3,quarantinePastSecs=77760000,quarantineFutureSecs=2592000,maxSliceSize=131072,serviceMetaPeriod=25,partialServiceMetaPeriod=0,throttleCheckPeriod=15,homePath_maxDataSizeBytes=0,coldPath_maxDataSizeBytes=0,compressionType=gzip,lz4BlockSize=65536,compressionLevel=-1,fsyncInterval=18446744073709551615,maxBloomBackfillBucketAge_secs=2592000,enableOnlineBucketRepair=true,enableDataIntegrityControl=false,maxUnreplicatedMsecWithAcks=60000,maxUnreplacatedMsecNoAcks=300000,alwaysBloomBackfill=false,minStreamGroupQueueSize=2000,streamingTargetTsidxSyncPeriodMsec=5000,repFactor=0,hotBucketTimeRefreshInterval=10,enableTsidxReduction=0,tsidxReductionCheckPeriodInSec=600,timePeriodInSecBeforeTsidxReduction=604800]' isSlave=false 08-20-2019 14:24:02.401 +0100 INFO IndexWriter - openDatabases complete currentId=499 idx=_introspection 08-20-2019 14:24:03.311 +0100 INFO IndexWriter - idx=_introspection Creating hot bucket=hot_v1_499, given event timestamped=1566307442 08-20-2019 14:24:03.311 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_introspection~499~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A' 08-20-2019 14:24:03.415 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).' 08-20-2019 14:24:04.042 +0100 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_introspection~498~6ED11580-5140-4ED9-BA6C-06B0C3FC8D1A 08-20-2019 14:24:04.347 +0100 INFO DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_introspection/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1' No visible errors being called out. ... View more

That didn't find anything. I checked _audit etc too. It's a weird one. Its only these 4 indexes. ... View more

I checked /opt/splunk/etc/system/local/indexes.conf and it only had an entry in there for the _introspection index. I took a copy of the file and rm ed the local/indexes.conf and restarted splunk. Running btool again I checked the output with a known good output from another indexer which is working fine for the _introspection index and they are identical but the index remains disabled. ... View more

Running the debug command gave the following output does look like the symptoms you though? splunk@spr-splunk-idx01 /opt/splunk/bin> ./splunk cmd btool indexes list --debug _introspection /opt/splunk/etc/system/local/indexes.conf [_introspection] /opt/splunk/etc/system/default/indexes.conf assureUTF8 = false /opt/splunk/etc/system/local/indexes.conf bucketRebuildMemoryHint = 0 /opt/splunk/etc/system/default/indexes.conf coldPath = $SPLUNK_DB/_introspection/colddb /opt/splunk/etc/system/default/indexes.conf coldPath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf coldToFrozenDir = /opt/splunk/etc/system/default/indexes.conf coldToFrozenScript = /opt/splunk/etc/system/local/indexes.conf compressRawdata = 1 /opt/splunk/etc/system/default/indexes.conf defaultDatabase = main /opt/splunk/etc/system/local/indexes.conf enableDataIntegrityControl = 0 /opt/splunk/etc/system/local/indexes.conf enableOnlineBucketRepair = 1 /opt/splunk/etc/system/default/indexes.conf enableRealtimeSearch = true /opt/splunk/etc/system/local/indexes.conf enableTsidxReduction = 0 /opt/splunk/etc/system/local/indexes.conf frozenTimePeriodInSecs = 1209600 /opt/splunk/etc/system/default/indexes.conf homePath = $SPLUNK_DB/_introspection/db /opt/splunk/etc/system/default/indexes.conf homePath.maxDataSizeMB = 0 /opt/splunk/etc/system/default/indexes.conf hotBucketTimeRefreshInterval = 10 /opt/splunk/etc/system/default/indexes.conf indexThreads = auto /opt/splunk/etc/system/default/indexes.conf journalCompression = gzip /opt/splunk/etc/system/default/indexes.conf maxBloomBackfillBucketAge = 30d /opt/splunk/etc/system/default/indexes.conf maxBucketSizeCacheEntries = 0 /opt/splunk/etc/system/default/indexes.conf maxConcurrentOptimizes = 6 /opt/splunk/etc/system/default/indexes.conf maxDataSize = 1024 /opt/splunk/etc/system/default/indexes.conf maxHotBuckets = 3 /opt/splunk/etc/system/default/indexes.conf maxHotIdleSecs = 0 /opt/splunk/etc/system/default/indexes.conf maxHotSpanSecs = 7776000 /opt/splunk/etc/system/default/indexes.conf maxMemMB = 5 /opt/splunk/etc/system/default/indexes.conf maxMetaEntries = 1000000 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroups = 8 /opt/splunk/etc/system/default/indexes.conf maxRunningProcessGroupsLowPriority = 1 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedNoAcks = 300 /opt/splunk/etc/system/default/indexes.conf maxTimeUnreplicatedWithAcks = 60 /opt/splunk/etc/system/default/indexes.conf maxTotalDataSizeMB = 500000 /opt/splunk/etc/system/default/indexes.conf maxWarmDBCount = 300 /opt/splunk/etc/system/default/indexes.conf memPoolMB = auto /opt/splunk/etc/system/default/indexes.conf minRawFileSyncSecs = disable /opt/splunk/etc/system/default/indexes.conf minStreamGroupQueueSize = 2000 /opt/splunk/etc/system/default/indexes.conf partialServiceMetaPeriod = 0 /opt/splunk/etc/system/default/indexes.conf processTrackerServiceInterval = 1 /opt/splunk/etc/system/default/indexes.conf quarantineFutureSecs = 2592000 /opt/splunk/etc/system/default/indexes.conf quarantinePastSecs = 77760000 /opt/splunk/etc/system/default/indexes.conf rawChunkSizeBytes = 131072 /opt/splunk/etc/system/default/indexes.conf repFactor = 0 /opt/splunk/etc/system/default/indexes.conf rotatePeriodInSecs = 60 /opt/splunk/etc/system/default/indexes.conf serviceMetaPeriod = 25 /opt/splunk/etc/system/default/indexes.conf serviceOnlyAsNeeded = true /opt/splunk/etc/system/default/indexes.conf serviceSubtaskTimingPeriod = 30 /opt/splunk/etc/system/default/indexes.conf streamingTargetTsidxSyncPeriodMsec = 5000 /opt/splunk/etc/system/default/indexes.conf suppressBannerList = /opt/splunk/etc/system/default/indexes.conf sync = 0 /opt/splunk/etc/system/local/indexes.conf syncMeta = 1 /opt/splunk/etc/system/default/indexes.conf thawedPath = $SPLUNK_DB/_introspection/thaweddb /opt/splunk/etc/system/default/indexes.conf throttleCheckPeriod = 15 /opt/splunk/etc/system/default/indexes.conf timePeriodInSecBeforeTsidxReduction = 604800 /opt/splunk/etc/system/default/indexes.conf tsidxReductionCheckPeriodInSec = 600 /opt/splunk/etc/system/default/indexes.conf tstatsHomePath = volume:_splunk_summaries/$_index_name/datamodel_summary /opt/splunk/etc/system/default/indexes.conf warmToColdScript = ... View more

Not encrypting data at all but showing reference values rather than display values. Now if I go only figure out how to get splunk integration to fix this. Reference Snow wiki look for Direct Web Services section 5 for detail. Can't include link no karma. ... View more