We're running an indexer cluster on 6.5.1. We found error in splunkd.log on one of the search peers:
12-06-2016 15:32:53.654 +0800 ERROR BucketReplicator - Replication failed due to open failure: file=/opt/splunk/var/lib/splunk/_internaldb/db/db_1480966426_1480764958_87_BF4B1947-4FB6-4464-BD62-299457B51B72/1480941784-1480764958-4821280532600088189.tsidx error='No such file or directory'
2-06-2016 15:34:01.061 +0800 ERROR BucketReplicator - Replication failed due to open failure: file=/mnt/security/db_1481009551_1480929735_9_BF4B1947-4FB6-4464-BD62-299457B51B72/1481007010-1481003314-4825555822462713745.tsidx error='No such file or directory'
Seems some tsidx files are 'lost'. There are other tsidx files in the same bucket. I've no idea what happened. Would anyone please help? Thanks.
Besides, a cold bucket folder for heavy index looks like following :
The first (quick) answer is that yes, buckets have their directory name to begin with "rb_" when they're replicated.
As for the missing TSIDX files, it may be possible to rebuild the bucket. From the CLI, you'd use something like splunk rebuild db_1479686070_1479451778_0_BF4B1947-4FB6-4464-BD62-299457B51B72. This builds the TSIDX files (and *.data files - metadata) from the raw data journal, all afresh.