You can replicate across WAN using our Splunk 6.1 Multisite clustering feature. ... View more

Do you know the server named i11_lx12mx is up and running? According to this CLI output that machine might be down ... View more

In Splunk 6 it is possible to search from SH both clustered indexers and non-clustered indexer. More info is here http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Configurethesearchhead#Search_across_both_clustered_and_non-clustered_search_peers ... View more

Got it. Even if only one indexer is available, the data will continue to be available and searchable. The optimization the blog post talks about is the order in which we fix indexes and commit generations. ... View more

In Splunk 6 you can have a SH search both clustered peers and one-off indexers. See here for more info http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Configurethesearchhead#Search_across_both_clustered_and_non-clustered_search_peers ... View more

I'm not sure i entirely understand the question. Are you wondering if it is possible to use single SH to search a clustered peers and one off non-clustered indexer? If so, then this is possible in v6. Please refer here http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Configurethesearchhead#Search_across_both_clustered_and_non-clustered_search_peers ... View more

For example, if I have two indexers, and have set an index replication & searchability factor of two, this won't actually work as expected No, If your replication policy is set to 2 and you have 2 indexers available, then your policy is already met, so users will be able to access and search the data. ... View more

ICSSupport, Are you getting the same error message? It means that the directory _cluster has restricted permission. Could you check the permission settings of that directory? ... View more

Agreed. Cluster Master HA is on our roadmap. ... View more

Could you check the permission on the _cluster directory? ... View more

Splunk compresses the data before storing it on disk. It also need to build search files (TSIDX) on top of the raw data to speed up searching. The following blog post talks about storage requirements in clustering. http://blogs.splunk.com/2013/01/31/disk-space-estimator-for-index-replication/ ... View more

Cluster master keeps only the meta data, so in order to bring up secondary master we need some basic static files like server.conf. All other information such as indexes, buckets etc can be dynamically rebuild by secondary master. More on this is documented here http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/Configurethemaster#Configure_a_stand-by_master Dolxor, I'm surprised that it took 4 - 6 hours for the cluster master to catch up. If all the peers are available and you just did a cluster master restart, it should get back to normal state in few mins. If you notice this again, please open a support case so we can get to the bottom of the issue ... View more

Now we forwarded events to single server and when primary indexer failed how does the replica copy of index is made primary? The cluster master receives the heart beats from indexers. As soon as the heart_beat_time_outinterval is passed, then the master declares the indexer is dead and reassigns the primary to the secondary indexer. ... View more

Hi Craig, Do you have enough number of peers to meet your replication and searchable policies? One reason it could say that it isn't searchable is some of the policies may not be met with current number of peers. We are in the process of refining the error messages to clearly distinguish between when the data is searchable (but with reduced number of copies) vs not completely searchable at all. Hope it helps Mustafa ... View more

You are asking if search head and peer can be run from the same machine under clustering. No, this is not supported combination. You can run search head on the same instance as master, though. BUT, from high availability standpoint, it is always recommended to run each component of cluster in a separate server. ... View more

The master retains the list of all peers connected to it and keeps the list for ever. That's why even after you removed the peer, it still lists them with the status of 'Down'. One workaround to clear this list is to restart the master. ... View more

The main reason we recommend to run each component of clustering (master, peer, search head) on separate server is because we don't want the failure of one component affects the other and bring down the entire cluster. ... View more

By default, all your pre-5.0 data is immediately searchable after enabling clustering, but it's not replicated. The reason these are not replicated is because we don't want to saturate your network with all replication activities when you enable clustering. One workaround is to have the old data to age out. Generally users retain the data in splunk for 90 days and then it ages out. So technically you have a single copy of the data for 90 days and all new 5.0 data will have required number of replicated copies. ... View more

That's right. Index replication will replicate all the indexes. ... View more

The recommended approach is to use replication master to control all the indexers. You can use option #2 as well, but you need to make sure that the DS doesn't restart replication master after pushing the configs. ... View more

As of Splunk 5.0, the recommended approach is Index Replication. More info http://docs.splunk.com/Documentation/Splunk/5.0/Indexer/Aboutclusters ... View more