Ssytem Indexes All Disabled

[samhodgson]

Hi,

Not sure when this occurred exactly however all of the indexes with an _ prefix are currently disabled on my indexer (non clustered distributed environment, 1 indexer + 1sh). I did reduce the size of the _internal index a while back which may be related, I have since changed this back and restarted to no avail.

splunkd.log does not show any related warnings or errors on restart as far as i can see. see below for end of splunkd.log after restart.

The indexes.conf does not specify any disabled params on any of the indexes, how can i re-enable these indexes?

07-18-2018 11:48:36.338 +0100 INFO ProcessTracker - (child_12__Fsck) Fsck - (bloomfilter only) Rebuild for bucket='/opt/splunk/var/lib/splunk/_internaldb/db/db_1531911675_1531910532_8926' took 42.81 milliseconds
07-18-2018 11:48:37.213 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
07-18-2018 11:48:37.214 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
07-18-2018 11:48:38.176 +0100 INFO IndexerIf - Asked to add or update bucket manifest values, bid=_internal~8926~620B4469-3CF8-4AF9-B52F-F77683DD529A
07-18-2018 11:48:38.205 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=1 . Reason='Updating manifest: bucketUpdates=1'
07-18-2018 11:48:38.205 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db
07-18-2018 11:48:40.896 +0100 INFO IndexWriter - Creating hot bucket=hot_v1_8927, idx=_internal, event timestamp=1531910771, reason="suitable bucket not found, number of hot buckets=0, max=3"
07-18-2018 11:48:40.896 +0100 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason='Adding bucket, bid=_internal~8927~620B4469-3CF8-4AF9-B52F-F77683DD529A'
07-18-2018 11:48:40.897 +0100 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_internaldb/db

[dkeck]

Please accept if this helped

[dkeck]

hm wierd, but I had the same problem today.

Workaround was to specify the disabled=false in system/local (but should work with any app). I still don´t see why it was disabled in the first place.

Someone restarted splunk with root(even though there is a "splunk" user) a couple of times, maybe thats the reason..?!

[dkeck]

Check this answer:

https://answers.splunk.com/answers/30986/why-is-my-index-disabled.html

might help.

[samhodgson]

Thanks dkeck,

I did look at this and cant see any duplicate buckets, also im not seeing any error in splunkd.log on restart and it seems to be all system buckets as opposed to just _internal.