Knowledge Management

Why does this error "('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays" causes KVstore to stop working?

mgaraventa_splu
Splunk Employee
Splunk Employee

Hi all, I’m experiencing an unclear issue with KVstore (Splunk 6.5.6). I’m leveraging field acceleration within KVstore collections.conf like this:

[TestCollection] 
accelerated_fields.myacceleration_Field1 = {"Field1”: 1} 
accelerated_fields.myacceleration_Field2 = {"Field2”: 1} 
accelerated_fields.myacceleration_Comp = {"Field1”: 1, "Field2”: 1} 

with this transforms.conf:

[MG_Collection] 
collection = TestCollection 
external_type = kvstore 
fields_list = Field1,Field2

and suddenly I am receiving these KVstore failures in splunkd.log:

07-30-2018 17:21:00.993 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:02.007 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:03.016 +0100 ERROR KVStorageProvider - An error occurred during the last operation ('createIndex', domain: '5', code: '10088'): exception: cannot index parallel arrays [Field2] [Field1] 
07-30-2018 17:21:03.017 +0100 ERROR KVStoreConfigurationProvider - Error syncing with KV Store, Initialization failed. 
07-30-2018 17:21:03.017 +0100 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. Failed to sync collection configurations. See splunkd.log for details.

I’m feeding the collection via SPL and I can’t understand why this is happening. The workarounds I have found to temporarily resolve the KVstore issue are:

A. transfer the SH cluster captain to the KVstore captain node: 

./splunk transfer shcluster-captain -mgmt_uri https://kvstorecaptain:8089 

B. perform a resync of the KVstore from the (KVstore/SH) captain: 

./splunk resync kvstore

Could you please help me to understand what is going wrong here?

1 Solution

mgaraventa_splu
Splunk Employee
Splunk Employee

What is happening here is that you are hitting a mongodb limitation, as described here:

https://docs.mongodb.com/manual/core/index-multikey/#limitations

This happens if more than one to-be-indexed field in the collection is an array. This potentially can happen if you first define a collection without any accelerated fields and then feed the collection with some SPL creating 2 or more multivalue fields like:

| makeresults 
| eval Field1="andy, matteo, john, brian", Field2="greg, daviann, zofnat, devon" 
| makemv delim="," Field1 
| makemv delim="," Field2 
| outputlookup MG_Collection append=t 

Once you define the field acceleration within collections.conf and then restart, you will hit the problem. If instead you define straight away the field acceleration in collections.conf and you attempt to save 2 or more multivalue fields within that KVstore collection, the search will fail and not store the data.

Hope this helps.

View solution in original post

mgaraventa_splu
Splunk Employee
Splunk Employee

What is happening here is that you are hitting a mongodb limitation, as described here:

https://docs.mongodb.com/manual/core/index-multikey/#limitations

This happens if more than one to-be-indexed field in the collection is an array. This potentially can happen if you first define a collection without any accelerated fields and then feed the collection with some SPL creating 2 or more multivalue fields like:

| makeresults 
| eval Field1="andy, matteo, john, brian", Field2="greg, daviann, zofnat, devon" 
| makemv delim="," Field1 
| makemv delim="," Field2 
| outputlookup MG_Collection append=t 

Once you define the field acceleration within collections.conf and then restart, you will hit the problem. If instead you define straight away the field acceleration in collections.conf and you attempt to save 2 or more multivalue fields within that KVstore collection, the search will fail and not store the data.

Hope this helps.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...