topic index replication unable to complete full index data in Splunk Dev

index replication unable to complete full index data

kplem — Tue, 12 Mar 2013 08:56:49 GMT

i have managed to set up cluster & index replication for 2 nodes for testing purpose. I have managed to replicate the index over to other splunk instance by editing the indexes.conf. However, the indexer seems to unable to replicate the full index. it only managed to index a very small amount of data over. Is there something missing. In short, i can see the replicated index on other splunk instance but the data is incomplete.

Re: index replication unable to complete full index data

gkanapathy — Tue, 12 Mar 2013 11:20:45 GMT

Why do you believe the data is incomplete? How are you looking? If you're simply looking at the file size of the buckets, it's entirely expected that a non-searchable replica will be considerably smaller (1/3 or less) than the size of a searchable replica. A searchable replica should be approximately (but not necessarily exactly) the same size on both systems.

Re: index replication unable to complete full index data

jdunlea_splunk — Tue, 12 Mar 2013 15:34:16 GMT

Also, remember that if you are setting up a second indexer to engage in replication with an existing indexer, then the existing data will NOT be replicated. You will only replicate data that was indexed AFTER you enabled replication.

Re: index replication unable to complete full index data

kplem — Wed, 13 Mar 2013 01:06:22 GMT

I think that is what happen to my set up. it only replicates what the peer has after the cluster set up. In that case, is there a way to allow the new index to have full replication of the data of its peer? Also does the replication occurs all the time or a time can be scheduled?

Re: index replication unable to complete full index data

jdunlea_splunk — Wed, 13 Mar 2013 14:29:38 GMT

When you add a new "index", (and the repFactor attribute is set to "auto" in indexes.conf) then all data that enters that index will be replicated. If you add a new "indexER" (Note difference between "index" and "indexer") then all data in that indexer will be replicated, if you have indeed set it up as an indexer in the cluster pool. Replication happens all the time for every 64bit chunks of data (as far as I know). Hope that helps.

Re: index replication unable to complete full index data

Steve_G_ — Wed, 13 Mar 2013 17:33:52 GMT

For more information on what happens to data that was already indexed prior to the indexer getting converted to a cluster peer (aka, "legacy data"), see http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/Migratenon-clusteredindexerstoaclusteredenvironment#Manage_legacy_data