Knowledge Management
Highlighted

recommended way to rename a kvstore collection that is not empty ?

Explorer

Dear all,

I am pretty new with KVstore, REST API and Python SDK, therefore my question might be trivial for an expert, but after some hours spent on answers.splunk.com I still don't get a real solution to that.

We are using Splunk Enterprise 8.0.0 and by reading the "Endpoints reference list": https://docs.splunk.com/Documentation/Splunk/8.0.0/RESTREF/RESTlist

I see that the REST-API allows for collections creation, adding items, updating items (in the sense of full updates), collections delete.

I couldn't find however anything about collections renaming - how should this work ?

The direct solution of gathering the data, creating another collection, pushing the data into the new collection then deleting the old collection does not seem a good choice when working with large collections (>= 10.000.000 items).

So the question is, what is the splunk way to rename an existing collection ? I simply refuse to think that splunk does not offer an interface method to realise this.

thx,
marius

Labels (1)
0 Karma
Highlighted

Re: recommended way to rename a kvstore collection that is not empty ?

SplunkTrust
SplunkTrust

Likely the safest way is make the new collection like the old one, configure the transforms lookup definitions. inputlookup and outputlookup to copy and then remove the old collection.

0 Karma
Highlighted

Re: recommended way to rename a kvstore collection that is not empty ?

Explorer

well, this is exactly the copy and delete way, which should be simply the latest alternative if nothing better exists.

Splunkers: I cannot think that there is no intelligent way of renaming a collection!

0 Karma
Highlighted

Re: recommended way to rename a kvstore collection that is not empty ?

Contributor

Hi Mhergh,

Splunk uses Mongo DB to provide the KV store functionality. Splunk uses an inbuilt custom Mongo instance to serve this purpose.I believe this is the reason by which we don't have an option to rename KV store collection.
We could have renamed the Mongo collection name directly if we were able to access the Mongo storage within splunk, however Splunk does not provide an option to do so.

However we could try and test a workaround.I am NOT sure whether this will work - so do it only on your TEST instance and NOT on prod. Ideally recommended to create a sample KV store to test this out.

Step1) Connect to your Splunk Linux Box

Step2) Get all the reference of your collection name.You could use the grep command in putty to get that.

grep  "your_kv_store_name" $(find /opt/splunk/etc/apps  -type f -name '*.conf')       
grep  "your_kv_store_name" $(find /opt/splunk/etc  -type f -name '*.conf')
grep  "your_kv_store_name" $(find /opt/splunk/ -type f -name '*.conf')

Step 3) Rename all the collection name reference.

Step 4 ) Restart splunk

Kindly let us know whether this works.

Highlighted

Re: recommended way to rename a kvstore collection that is not empty ?

Explorer

Unfortunately it didn't work at all. After "renaming" as recommended, the renamed collections were present but empty; the previous collections have been dropped.

I didn't expect to work either: how should know splunk by just changing the collection name in the config files that he has to 're-link' to the previous collection ?

However, MongoDB supports collection renaming except for sharded collections. See here: https://docs.mongodb.com/manual/reference/command/renameCollection/#dbcmd.renameCollection?searchPro...

So, probably splunk automatically employs sharded collections by default to allow for the collection to be distributed across the head-searches oder so (I assume) and this renders to the inability to elegantly rename collections.

You may please close this topic, now I am convinced that only the 'brut-force' solution - to copy over - is possible because of MongoDB.

0 Karma
Highlighted

Re: recommended way to rename a kvstore collection that is not empty ?

Contributor

Hi mhergh,

Thanks for confirming.
I was just thinking of the possibility of any re-balancing in the KV stores which may push the new config changes to MongoDB.

Yes,Mongo db allows to rename collection - but that functionality is not exposed and also, we are not allowed to connect to Mongo client.

So our only option is to copy over the data.

It will be nice if you can upvote/accept the answer - so that other splunkers can also refer.

Happy Splunking!!

0 Karma