topic Re: How to migrate back the data stored in Smartstore to persistent disk in Splunk Enterprise

How to migrate back the data stored in Smartstore to persistent disk

manikanta461 — Mon, 22 Jan 2024 13:45:54 GMT

Hello All,

Recently we have migrated all our indexes to Splunk Smartstore with our remote storage being Azure blob.
After that we noticed several problems with our environment.

Buckets being stuck in fixup state more often.
Indexing queues being full (No major spike in data indexation).
Huge increase in number of buckets.

And the list goes on.

We are considering to revert back to the persistent disk for data storage, however, looking at the Splunk documentation, it is not possible to revert back an index configured with Splunk Smartstore perisitent disk. But, I'm looking at a way, if it would be still possible to do it, because of the above issues, the search performance is abysmal.

We have around 6 indexers and each indexer has around 800k buckets and the current data on remote storage (Smartstore) is 50 TB.

Are there any ways to migrate back to persistent disk? Looking forward to any gray methods to try out as well.

Thanks

Re: How to migrate back the data stored in Smartstore to persistent disk

richgalloway — Mon, 22 Jan 2024 15:14:40 GMT

The documentation is correct. Once you go to SmartStore you can't go back; anything else would be a Science Experiment.

Switching to SmartStore (S2) should not have caused the problems you listed. Search performance can be affected if the S2 cache is too small or users have a tendency to search over more than 30 days.

Is SmartStore in the same environment as your indexers? Using a cloud S2 with on-prem indexers is likely cause problems and be expensive.

Re: How to migrate back the data stored in Smartstore to persistent disk

manikanta461 — Mon, 22 Jan 2024 15:17:17 GMT

Thanks for the answer, at least the indexers are in the same environment as the smartstore and not on-prem.

Re: How to migrate back the data stored in Smartstore to persistent disk

linhmai_bne — Tue, 23 Jan 2024 04:17:50 GMT

I agreed that we should have option to move data from SmartStore back to local storage.

Re: How to migrate back the data stored in Smartstore to persistent disk

isoutamo — Thu, 25 Jan 2024 11:20:05 GMT

Your best (and probably only) option is to connect your splunk account manager and ask PS support for this case. At least they could do an analyse about your environment and check is is what it required. They also could create a plan how this situation could fixed asap.

r. Ismo

Re: How to migrate back the data stored in Smartstore to persistent disk

AkshayRayapudi — Tue, 16 Apr 2024 12:19:38 GMT

Hi @manikanta461 , We've migrated one of our high volume index to smart store and facing the exact same issues as you've described in your post. Could you please tell me how you resolved this issue? And, if you've performed a roll-back, what steps should be taken to minimize data loss/impact?