Splunk Dev

Splunk Enterprise - How do I move indexed data from a cluster of indexers to a single instance?

scottrunyon
Contributor

Due to the amount of bugs that I have been running into since moving to a clustered environment, I want to return to a single instance.

I found http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Moveanindex and it looks like all I have to do is move the \Splunk\var\lib\splunk directory to the new system. But nothing is ever that easy.

I am worried that since the buckets are duplicated, i will end up with too much data and Splunk will not know how to process that extra data.

I am also ok with some telling me how to roll back my clustered systems into a single system.

Thanks.

0 Karma
1 Solution

scottrunyon
Contributor

I did move the data from \splunk\var\lib\splunk\ from the old systems to the new system. I had to modify the manifest.csv and the and .bucketManifest files with the folder names of the old data. After a restart, the data showed up!

View solution in original post

scottrunyon
Contributor

I did move the data from \splunk\var\lib\splunk\ from the old systems to the new system. I had to modify the manifest.csv and the and .bucketManifest files with the folder names of the old data. After a restart, the data showed up!

Krishnagrandhi
Explorer

Did rsync from 8 indexers to single indexer. merge all .bucketManifest and manifest.csv from all indexers. Restart single splunk instance. Event count is similar to 8 indexers total event count.
use the following to compate bucket count.
index="" | eval BID = replace(_cd, "(\d+):\d+", "\1") | top BID

0 Karma

woodcock
Esteemed Legend

Once you go clustered, you cannot go unclustered or you will lose your data. It exists in a different format than the "original" buckets do. What you can do is go to replication factor of 0 and this turns off all the clustering but still allows you to keep your existing (old) data. Otherwise you will have to abandon your old data.

Actually, on second thought, this might not be true, but it probably is. I know it is true the other way around (clustered indexers cannot deal with unclustered index buckets), but the reverse might be possible.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hey Scottrunyon, sorry to hear you are having challenges with clustering. If you have a support contract, please do reach out to our support teams so they can help validate your claims of "bugs".

Can you tell us more about your environment?

How many indexers do you have? What was your rep and search factor?

Are you hoping to collapse the environment into one indexer? Or simply revert to non clustered indexers?

Moving data is as simple as the docs advise, but technically you only need one copy of each bucket...so doing that will depend on the cluster you have set up.

Let us know a bit more about what you have done and we can try and help you out!

- MattyMo

scottrunyon
Contributor

I have a Master, 2 indexer cluster, 1 Enterprise Security dedicated system and 1 search head.

I want to drop ES and go back to a single system.

As for Support, that is where I got the answers about bugs. Specifically, the response was that bugs SPL-140260 and SPL-140831 were the problem and Support suggested that I downgrade to version 6.4.x. Of course, there is no supported way to downgrade.

I am spending too much time playing whack-a-mole trying to keep these systems up. There are too many crashes, too much overhead traffic, etc.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Again, sorry to hear you are running into issues.

I would recommend consulting your account team to ensure you get the best path forward for your particular environment.

How much data are you ingesting daily?

You could simply stand up a standalone box and cut over to it, then roll your buckets from the cluster to frozen and then thaw those on the standalone...

Seems a shame to go that route tho, especially if you have purchased ES....

But hey...running Splunk, especially distributed with ES ain't easy, thats for sure, so i can sympathize with your wants to go back to good ol' standalone...

- MattyMo
0 Karma

scottrunyon
Contributor

We index about 6GB of data on weekdays with only 3GB on the weekends.

When you say "roll your buckets from the cluster to frozen and then thaw those on the standalone..." how would I do that?

As for ES, we use it for is get the treat feeds.

Regards,

Scott

0 Karma

mattymo
Splunk Employee
Splunk Employee

http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Automatearchiving
http://docs.splunk.com/Documentation/Splunk/6.6.1/Indexer/Restorearchiveddata

Because you only have 2 indexers is shouldn't be too hard for you to go through the data and ensure you only move 1 copy to the standalone.

- MattyMo
0 Karma

scottrunyon
Contributor

Looking at the doc, it appears to me that only data that would be removed (cold to frozen) is instead being archived. It doesn't look like all the indexed data wouild be moved using this process.

I did a test run of coping the colddb, datamodel_summary_db and thaweddb directories of an index plus the .dat file in /var/lib/splunk location and it looks like the indexed data is up to data and searchable. I am going that route, Fingers crossed.

0 Karma

mattymo
Splunk Employee
Splunk Employee

You would set your index retention to force the data to roll to frozen, but hey, if copying the data works, i say do it!

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...