Splunk Dev

index replication unable to complete full index data

kplem
New Member

i have managed to set up cluster & index replication for 2 nodes for testing purpose. I have managed to replicate the index over to other splunk instance by editing the indexes.conf. However, the indexer seems to unable to replicate the full index. it only managed to index a very small amount of data over. Is there something missing. In short, i can see the replicated index on other splunk instance but the data is incomplete.

Tags (1)
0 Karma

jdunlea_splunk
Splunk Employee
Splunk Employee

Also, remember that if you are setting up a second indexer to engage in replication with an existing indexer, then the existing data will NOT be replicated. You will only replicate data that was indexed AFTER you enabled replication.

0 Karma

Steve_G_
Splunk Employee
Splunk Employee

For more information on what happens to data that was already indexed prior to the indexer getting converted to a cluster peer (aka, "legacy data"), see http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Migratenon-clusteredindexerstoaclustereden...

jdunlea_splunk
Splunk Employee
Splunk Employee

When you add a new "index", (and the repFactor attribute is set to "auto" in indexes.conf) then all data that enters that index will be replicated. If you add a new "indexER" (Note difference between "index" and "indexer") then all data in that indexer will be replicated, if you have indeed set it up as an indexer in the cluster pool. Replication happens all the time for every 64bit chunks of data (as far as I know). Hope that helps.

0 Karma

kplem
New Member

I think that is what happen to my set up. it only replicates what the peer has after the cluster set up. In that case, is there a way to allow the new index to have full replication of the data of its peer? Also does the replication occurs all the time or a time can be scheduled?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Why do you believe the data is incomplete? How are you looking? If you're simply looking at the file size of the buckets, it's entirely expected that a non-searchable replica will be considerably smaller (1/3 or less) than the size of a searchable replica. A searchable replica should be approximately (but not necessarily exactly) the same size on both systems.

0 Karma
Get Updates on the Splunk Community!

New Dates, New City: Save the Date for .conf25!

Wake up, babe! New .conf25 dates AND location just dropped!! That's right, this year, .conf25 is taking place ...

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...