Error STmgr - dir unexpected

[MUmair_DOI]

I have two IDX pointed to a SH a couple of weeks an error started flooding in from Splunkd. It looks to be for metrics.log file, but I cannot seem understand what the error is and have not been able to figure out a solution by searching the community forums. 

Essentially, the following errors continue to come in about 1000 errors an hour or so. It was only coming from 1 IDX at first, but now its coming from bother IDXs. 

Sample errors:

01-25-2021 16:50:16.946 +0000 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_49' unexpected rc=-104 (kw= sourcetype::splunk_metrics_log, len=31) warm_rc[0,2] from st_txn_put

01-25-2021 16:50:16.946 +0000 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_49' unexpected rc=-104 (kw= host::iinabqlvtsplidx2, len=23) warm_rc[0,2] from st_txn_put

01-25-2021 16:50:16.946 +0000 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_49' unexpected rc=-104 (kw= source::/opt/splunk/var/log/introspection/kvstore.log, len=54) warm_rc[0,2] from st_txn_put

01-25-2021 16:50:16.946 +0000 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_49' unexpected rc=-104 (kw=_catalog::spl.mlog.nullgroup.data.metrics.commands._mergeAuthzCollections.total|CN|O|component|data.$clusterTime.signature.hash.$binary|data.extra_info.note|data.host|data.mem.supported|data.metrics.repl.executor.networkInterface|data.metrics.repl.executor.shuttingDown|data.network.serviceExecutorTaskStats.executor|data.process|data.repl.electionId.$oid|data.repl.hosts|data.repl.ismaster|data.repl.me|data.repl.primary|data.repl.secondary|data.repl.setName|data.repl.tags.all|data.repl.tags.instance|data.security.SSLServerHasCertificateAuthority|data.security.SSLServerSubjectName|data.storageEngine.name|data.storageEngine.persistent|data.storageEngine.readOnly|data.storageEngine.supportsCommittedReads|data.tcmalloc.tcmalloc.formattedString|data.version|datetime|log_level, len=779) warm_rc[0,2] from st_txn_put

01-25-2021 16:50:16.946 +0000 ERROR STMgr - dir='/opt/splunk/var/lib/splunk/_metrics/db/hot_v1_49' unexpected rc=-104 (kw=_catalog::spl.mlog.nullgroup.data.globalLock.currentQueue.total|CN|O|component|data.$clusterTime.signature.hash.$binary|data.extra_info.note|data.host|data.mem.supported|data.metrics.repl.executor.networkInterface|data.metrics.repl.executor.shuttingDown|data.network.serviceExecutorTaskStats.executor|data.process|data.repl.electionId.$oid|data.repl.hosts|data.repl.ismaster|data.repl.me|data.repl.primary|data.repl.secondary|data.repl.setName|data.repl.tags.all|data.repl.tags.instance|data.security.SSLServerHasCertificateAuthority|data.security.SSLServerSubjectName|data.storageEngine.name|data.storageEngine.persistent|data.storageEngine.readOnly|data.storageEngine.supportsCommittedReads|data.tcmalloc.tcmalloc.formattedString|data.version|datetime|log_level, len=763) warm_rc[0,2] from st_txn_put

Basically the same error are repeating over and over again in a similar fasion.

[AlvaroFernandez]

Same here since the migration to v8.0.9. 

No solution found so far