Deployment Architecture

Missing data after index restore

swangertyler
Path Finder

I do not understand why, but I appear to have missing data. We have 4 indexers in our cluster and in our server.conf filer replication_factor=4. Stopped splunkd on everything. We backed up each index, on each indexer (each folder under "/opt/splunk/var/lib/splunk/" on each index got backed up). Performed the necessary maint on the cluster. Took everything we backed up on "old indexer1" and put it back on "new indexer1", started splunkd, and had it replicate across the indexers.

But when I search the data from a search head, using the SPL, I know there is stuff missing.

Beyond going through each of my archives, finding what buckets are missing, and restore them, what am I missing? I assumed since replication_factor=4, that meant I had duplicate copies of all of my data across all 4 indexers. What am I missing?

0 Karma

teunlaan
Contributor

1) How do you know stuff is missing?
2) are you missing a single index, or a certain time frame?
3) Why do you backup and restore data from 1 machine in your cluster? Wouldn't it be much simpelere too add a new machine and than remove the old one? Then splunk would take care of you repliaction.

Did you add you new server in the cluster pool?

0 Karma

swangertyler
Path Finder
  1. We have a test environment that AFAIK (I am now questioning) should be identical, and I had compared the two.
  2. Does not appear to be a single index. I have not identified if it is a certain timeframe or not. I can check.
  3. We needed to rename an index for reasons.
0 Karma

teunlaan
Contributor

oh you are renameing an index. So index=bla becomes index=testingbla ?
not sure if a cluster "likes" changing names of an index, and still can find the data

where did you place the "restored" data? did you place it in the thawed directory?
You could thy a rebuild, but you need too do this for every bucket !

0 Karma

swangertyler
Path Finder

"opt/splunk/var/lib/splunk" is where all of our indices live.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...