How to migrate back the data stored in Smartstore to persistent disk

[manikanta461]

Hello All,

Recently we have migrated all our indexes to Splunk Smartstore with our remote storage being Azure blob.
After that we noticed several problems with our environment.

• Buckets being stuck in fixup state more often.
• Indexing queues being full (No major spike in data indexation).
• Huge increase in number of buckets.

And the list goes on.

We are considering to revert back to the persistent disk for data storage, however, looking at the Splunk documentation, it is not possible to revert back an index configured with Splunk Smartstore perisitent disk. But, I'm looking at a way, if it would be still possible to do it, because of the above issues, the search performance is abysmal.

We have around 6 indexers and each indexer has around 800k buckets and the current data on remote storage (Smartstore) is 50 TB.

Are there any ways to migrate back to persistent disk? Looking forward to any gray methods to try out as well.

Thanks

[AkshayRayapudi]

Hi @manikanta461 , We've migrated one of our high volume index to smart store and facing the exact same issues as you've described in your post. Could you please tell me how you resolved this issue? And, if you've performed a roll-back, what steps should be taken to minimize data loss/impact?

[linhmai_bne]

I agreed that we should have option to move data from SmartStore back to local storage.

[richgalloway]

The documentation is correct.  Once you go to SmartStore you can't go back; anything else would be a Science Experiment.

Switching to SmartStore (S2) should not have caused the problems you listed. Search performance can be affected if the S2 cache is too small or users have a tendency to search over more than 30 days.

Is SmartStore in the same environment as your indexers?  Using a cloud S2 with on-prem indexers is likely cause problems and be expensive.

[manikanta461]

Thanks for the answer, at least the indexers are in the same environment as the smartstore and not on-prem.

[isoutamo]

Hi

Your best (and probably only) option is to connect your splunk account manager and ask PS support for this case. At least they could do an analyse about your environment and check is is what it required. They also could create a plan how this situation could fixed asap.

r. Ismo