Splunk Cluster: Index not Replicating

[fbustamantes]

Hello,

I have been trying to configure a little lab enviroment to test the replication functionality of Splunk 5 (currently we are using 5.0.2 in all hosts involved). We have set 1 host as master, and the other two with one search head instance and one indexer each. We set up a forwarder in a external host, and currently we are generating data using a script to generate log data. The forwarder is pointing to Peer 1.

We created a new index called 'rep_test' in indexes.conf, with 'repFactor=auto' and pushed it to the peers using the master (through the _cluster dir and using the 'splunk apply cluster-bundle' command). Everything worked fine, so once the index was created on both peers, we configured the forwarder to start sending data to the first peer (Peer 1). We tried searching for the data on both search heads an everything worked fine. We see the index and both peers in the master's cluster dashboard and data is coming in just fine. However, when we check on Peer 2 to see if data is getting replicated to the index, we are not seeing any changes. The only way to see changes is when we perform a restart of the peers from the master, but I guess it's not the idea to restart each time I want to replicate data.

The master's replication factor is set to 2.

Could you please help me find what am I missing? We have checked all the documentation, but there's nothing specific, and I'm not quite sure of what should I be looking for in splunkd.log or other logs that could guide me to know why it's not working.

Thanks in advance for your help,

Felipe.

[Jon_Webster · Accepted solution]

Hi Felipe:

Index replication works by copying any new cold buckets from the originating indexer to additional indexers. The receiving indexer will not show the data unless the originating indexer is down.

This is all controlled by the master node. It keeps a list of all primary index buckets, and which indexer is primary for each bucket, and therefore will search which buckets.

This is the starting point in the docs to see exactly how this works. Start here and read through all the linked topics:
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Basicconcepts

Best,
Jon

[highsplunker]

hi,

am i correct that the data is replicated (i.e. present on both indexers) and available for search on search heads, but not shown when search on them (indexers) ?

i have not read yet the whole page, honestly...

[fbustamantes]

Very useful answer, thanks you.

We tried putting down the main indexer and the second kept showing data through our search heads, just as you described.

Problem solved!

Thanks!