This is okay. It just means that the bucket has been frozen on all nodes and removed completely from the cluster so the cluster master no longer knows about the bucket which is good for our purposes. The problem happens if the bucket is frozen only on a subset of the peers and then the master is restarted and it forgets that the bucket was frozen and proceeds to fixup since at least one valid copy of it exists Things I would suggest: Do the freeze command above for all the previously frozen buckets in that one index. (Though, if it has only 3 buckets and it hasn't fixed itself up yet, this might not be the problem. But we should probably try it in any case.) Commit the next generation as indicated above. Check the UI to see how things stand. If that doesn't work, re-add all the peers using the command dxu_splunk mentioned. (You need to do it for all 3 indexers; do it one at a time and wait for the master dashboard UI to show that peer as UP before doing the next one.) Sometimes a state mismatch between the master and peers can cause this problem and the re-add would fix some of that. Lets see where things stand after the above. If we still have a problem, we might need to figure out which bucket is causing the problem and so on. There are trouble shooting endpoints to check which exist on 6.0.x and above, but am not sure of 5.0.x. We might have to do it differently on 5.0.x I'll set up a 5.0.x cluster and take a look. ... View more

nivedita_viswanath, That error might mean a network issue (or more precisely a consistent replication issue ) between the two nodes. See http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/Bucketreplicationissues#Network_issues_impede_bucket_replication Is the replication port configured properly on the two nodes. Are there any errors related to that in the splunkd logs? ... View more

Also, if it is just 3 buckets, you can probably let it be for a bit and it should fix itself up. ... View more

Can you try this from the cli: for i in cat /var/tmp/frozen_buckets ; do curl -d "" -k -u admin:changeme https://localhost:8089/services/cluster/master/buckets/$i/freeze; done (EDIT: there is a backtick around cat /var/tmp/frozen_buckets not sure if that shows up in the comment) and then: $SPLUNK_HOME/bin/splunk _internal call /cluster/master/control/directive/commit_generation -method POST You can use a file with just the bucket-ids from the index you care about or just do everything. Can you let me know how that goes. ... View more

zislin, Did this problem show up after you restarted the master? In 5.0.x, the cluster master did not track frozen buckets properly. After a restart, the master would then proceed to fix up buckets that were previously frozen. See here for more info: http://docs.splunk.com/Documentation/Splunk/5.0.4/Indexer/Upgradeacluster#Why_the_safe_restart_cluster_master_script_is_necessary. The UI in 5.0.x also reported the worst case always: ie if there was even one bucket with no searchable copy then it would report that index as having no searchable copies. The problem might just be caused by a subset of the buckets like these frozen buckets. When you restart the master, a similar procedure as mentioned in the link for upgrades is needed. I wonder if this could be the problem in your case. Since you've already restarted the master, you cannot use that script as it is anymore since the information is already lost. But we might still be able to recover by (1) just giving the cluster enough time or (2) using search to figure out the list of buckets to be fixed and then scripting it from there. Try this search on the master from the cli, to get a list of frozen buckets $SPLUNK_HOME/bin/splunk search 'index=_internal component=CMMaster "remove bucket" frozen=true | dedup bid | table bid' -preview 0 > /var/tmp/frozen_buckets and maybe also: grep my_index /var/tmp/frozen_buckets | wc -l to see how many such buckets show up for that index. That will tell us if this is the problem ... View more

Looks like that endpoint was added in 5.0.2 so that endpoint should work in 5.0.7 ... View more

samlaw, What does splunk status report? Were you able to access any REST endpoint (say https://hostname:mgmtPort/services/) via say the browser ... View more

Ed_alias, Hot buckets are replicated too. (The replication is not per-event but a certain slice of data.) See http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Howclusteredindexingworks for more information. Could you elaborate on what exactly was the issue? ... View more

R.Turk, How much data did you index? (The replication is not per-event but per some amount of data. I'm wondering if the replication never happened before the shutdown.) Also are you sending data via forwarder or local file via monitors? If forwarders, is your forwarder auto-lbing across all the peers? If the peer goes down without replicating the data, then the forwarder should just send the data to some other peer. But if you are indexing local files via monitors that won't happen. You should be using forwarders that have acks turned on and set to auto-lb across all those peers. ... View more

That could be the cause of your problem. Though the BucketReplicator errors on the originating node not on the target of the replication. When it says "Too many streaming errors" that's anything that causes the replication to fail. Do you see why the writes fail? Did they just timeout? Also when you say you checked the network adapter, what did you check? ... View more