Why does this error "('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays" causes KVstore to stop working?

[mgaraventa_splu]

Hi all, I’m experiencing an unclear issue with KVstore (Splunk 6.5.6). I’m leveraging field acceleration within KVstore collections.conf like this:

[TestCollection] 
accelerated_fields.myacceleration_Field1 = {"Field1”: 1} 
accelerated_fields.myacceleration_Field2 = {"Field2”: 1} 
accelerated_fields.myacceleration_Comp = {"Field1”: 1, "Field2”: 1} 

with this transforms.conf:

[MG_Collection] 
collection = TestCollection 
external_type = kvstore 
fields_list = Field1,Field2

and suddenly I am receiving these KVstore failures in splunkd.log:

07-30-2018 17:21:00.993 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:02.007 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:03.016 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:03.017 +0100 ERROR KVStoreConfigurationProvider - Error syncing with KV Store, Initialization failed. 
07-30-2018 17:21:03.017 +0100 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to sync collection configurations. See splunkd.log for details.

I’m feeding the collection via SPL and I can’t understand why this is happening. The workarounds I have found to temporarily resolve the KVstore issue are:

A. transfer the SH cluster captain to the KVstore captain node: 

./splunk transfer shcluster-captain -mgmt_uri https://kvstorecaptain:8089 

B. perform a resync of the KVstore from the (KVstore/SH) captain: 

./splunk resync kvstore

Could you please help me to understand what is going wrong here?

[mgaraventa_splu · Accepted solution]

What is happening here is that you are hitting a mongodb limitation, as described here:

https://docs.mongodb.com/manual/core/index-multikey/#limitations

This happens if more than one to-be-indexed field in the collection is an array. This potentially can happen if you first define a collection without any accelerated fields and then feed the collection with some SPL creating 2 or more multivalue fields like:

| makeresults 
| eval Field1="andy, matteo, john, brian", Field2="greg, daviann, zofnat, devon" 
| makemv delim="," Field1 
| makemv delim="," Field2 
| outputlookup MG_Collection append=t 

Once you define the field acceleration within collections.conf and then restart, you will hit the problem. If instead you define straight away the field acceleration in collections.conf and you attempt to save 2 or more multivalue fields within that KVstore collection, the search will fail and not store the data.

Hope this helps.