After upgrading to 5.01 we began receiving this error.
received event for unconfigured/disabled/deleted index='_audit' with source='source::audittrail' host='host::foo' sourcetype='sourcetype::audittrail' (1 missing total)
Looking at the Indexes I can see the _audit index is disabled with the current size of the file being 0MB.
I tried setting _audit to Enable but receive and error message that: One or more indexes could not be initialized and were automatically disabled, please see splunkd.log for more details
Looking the spunkd.log file this is what is reported:
11-30-2012 13:09:27.747 -0800 INFO IndexProcessor - reloading index config: request received
11-30-2012 13:09:27.749 -0800 INFO IndexProcessor - reloading index config: start
11-30-2012 13:09:27.749 -0800 INFO IndexProcessor - request state change from=RUN to=RECONFIGURING
11-30-2012 13:09:27.749 -0800 INFO IndexProcessor - Initializing: readonly=false reloading=true
11-30-2012 13:09:27.754 -0800 INFO IndexProcessor - Got a list of count=1 added, modified, or removed indexes
11-30-2012 13:09:27.755 -0800 INFO IndexProcessor - Reloading index config: shutdown subordinate threads, now restarting
11-30-2012 13:09:27.755 -0800 INFO IndexProcessor - indexes.conf - indexThreads param autotuned to=2
11-30-2012 13:09:27.755 -0800 INFO HotDBManager - idx=_audit Setting hot mgr params: maxHotSpanSecs=7776000 snapBucketTimespans=false maxHotBuckets=3 maxDataSizeBytes=786432000 quarantinePastSecs=77760000 quarantineFutureSecs=2592000
11-30-2012 13:09:27.755 -0800 INFO databasePartitionPolicy - idx=_audit Initialized with params='[300,60,188697600,,,,786432000,5,true,500000,5,5,false,3,0,_blocksignature,7776000,1000000,0,3,77760000,2592000,131072,25,0,15,0,0,-1,18446744073709551615,2592000,true,60000,300000,false]' isSlave=false needApplyDeleteJournal=false
11-30-2012 13:09:27.756 -0800 ERROR DatabaseDirectoryManager - idx=_audit bucket=hot_v1_0 Detected directory manually copied into its database, causing id conflicts [path1='C:\Program Files\Splunk\var\lib\splunk\audit\db\db_1326238803_1326231564_0' path2='C:\Program Files\Splunk\var\lib\splunk\audit\db\hot_v1_0'].
11-30-2012 13:09:27.756 -0800 ERROR DatabaseDirectoryManager - idx=_audit bucket=hot_v1_20 Detected directory manually copied into its database, causing id conflicts [path1='C:\Program Files\Splunk\var\lib\splunk\audit\db\db_1331855014_1331854207_20' path2='C:\Program Files\Splunk\var\lib\splunk\audit\db\hot_v1_20'].
11-30-2012 13:09:27.756 -0800 ERROR IndexProcessor - caught exception for idx=_audit during initialization: 'idx=_audit bucket=hot_v1_20 Detected directory manually copied into its database, causing id conflicts [path1='C:\Program Files\Splunk\var\lib\splunk\audit\db\db_1331855014_1331854207_20' path2='C:\Program Files\Splunk\var\lib\splunk\audit\db\hot_v1_20'].'.Disabling the index, please fix-up and run splunk enable index
11-30-2012 13:09:27.759 -0800 ERROR IndexProcessor - One or more indexes could not be initialized and were automatically disabled, please see splunkd.log for more details
11-30-2012 13:09:27.764 -0800 INFO IndexProcessor - request state change from=RECONFIGURING to=RUN
11-30-2012 13:09:27.764 -0800 INFO IndexProcessor - reloading index config: end
Any help to correct this would be appreciated.
Thank you
Doug
Check your indexes.conf file. You may have an index that got accidentally disabled and if that is the case, you'll see that error. If that is the case, just flip it from 1 to 0 and restart. That should take care of it.
If you see none of those, maybe you have a reference in inputs.conf or props.conf on a stanza for an index that you deleted. I'd check those as well.
Bucket collision. Did you ever get this resolved?