In order to consider your option it’s good idea to know about Bug# SPL-110192:Multi-site buckets should not be bonded to Originating Site Due to this Bug if you remove site3 and decommission the Peer on site 3, all the bucket that had Originating(created) on indexers of Site3 will continue to throw message “missing={site3:x}enough start targets=1”. This message is annoying but can be complete ignore entire data will still be searchable. These messages will eventually go away once these buckets age out and you will be back in state without this errors.So, when you move the indexer of site3 to site1 or site you- you will be better of re-installing the splunk instance and adding these as fresh. On the other side if you decide to keep the site3 configuration you will need to have at least one copy of bucket and you can use configuration like below- and stop forwarding any data to site 3 indexer- eventually over time when the data ages out – decommission site 3. site_replication_factor = origin:, site1: , site2: , site3:1, total:4 site_search_factor = origin:, site1:, site2:, site3:1 total:2 ... View more

The setting you are planning to apply will help reduce some amount of disk utilization. To make this change you need to follow following steps. 1)On the Cluster Master make changes to server.conf ( this change needs Cluster Masters restart ) 2) After that you can use the remove excess command as listed in http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/Removeextrabucketcopies 3) My advice will be to be selective - Do few index at a time and wait cluster to meet RF and SF and perform others. Here are Bugs around "remove excess-buckets" that you might be interested in. SPL-108023/ SPL-98101 :[Clustering] "remove excess-buckets" removes all buckets created under Multi-Site clustering if CM was moved from Multisite to Single Site. ( Reported on : 6.2.4, 6.2.6 and fixed on 6.2.7 and 6.3) SPL-90409 :remove excess buckets does not remove all the excess buckets it should and causes "fully searchable" criteria in UI to fail ( This is not a majore buck: remove-excess-buckets may cause all-searchable to become not-all-searchable, but it should quickly return to all-searchable) - fixed in 6.3 SPL-106614 :remove excess bucket doesn't remove summary files (this bug is still pending) SPL-90986:splunk list excess-buckets only lists 30 indexes (Reported on 6.1.3 and Resolved in 6.3) SPL-98007:Clicking "remove" button in the cluster management excess buckets page deletes all standalone buckets (Reported on 6.2.1, 6.2.2 and fixed in 6.2.3) In Splunk Version 6.3 a new Capability has been added that could help you manage such a situation better.Refer : https://answers.splunk.com/answers/331293/if-two-cluster-peers-in-an-indexer-cluster-are-get.html ... View more

Here are two options that worked with me… Option 1 : Use Splunk to Decrypt Password Encrypted by Splunk : Refer :- https://hurricanelabs.com/blog/decrypt-passwords-encrypted-by-splunk/ Option 2 : in this approach follow +Install the new instance of Search Head Cluster Member . (don’t start Splunk instance yet) +Copy $SPLUNK_HOME_EXISTING_SHC/etc/auth/splunk.secret from one of the existing Search Head cluster to New instances $SPLUNK_HOME_NEW_SHC/etc/auth +After that start splunk for the first time copy $SPLUNK_HOME_EXISTING_SHC/etc/system/local/server.conf stanza [shclustering] to $SPLUNK_HOME_NEW_SHC/etc/system/local/server.conf [shclustering] conf_deploy_fetch_url = https://DEPLOYER_URI:DEPLOYER_MGMT_PORT disabled = 0 mgmt_uri = https://:SPLUNK_HOME_NEW_SHC_MGMT_PORT pass4SymmKey = $1$XccY3P4= replication_factor = 1 id = B1D0A95D-4DB6-4111-A702-57EADDFFC932 +Restart the Splunk new Search Head cluster Member. +Follow the link to add it as Search Head Cluster member to exiting SHC group. ... View more

I just tested by deploying this on my Search Head Cluster and below is my experience . 1) Before deploying the app - I installed dnslookup on the deployer server. 2) After installing this App on the deployer -when I restarted it it throws the following message $SPLUNk_HOME/bin/splunk restart Stopping splunkd... Shutting down. Please wait, as this may take a few minutes. .. [ OK ] Stopping splunk helpers... [ OK ] Done. Splunk> See your world. Maybe wish you hadn't. Checking prerequisites... Checking http port [24400]: open Checking mgmt port [24401]: open Checking appserver port [127.0.0.1:24405]: open Checking kvstore port [24406]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _thefishbucket history main msad perfmon summary winevents Done Checking filesystem compatibility... Done Checking conf files for problems... Invalid key in stanza [install] in /opt/splunk/etc/apps/dnslookup/default/app.conf, line 10: author (value: Travis Freeland). Invalid key in stanza [install] in //opt/splunk/etc/apps/dnslookup/default/app.conf, line 11: description (value: dnslookup <forward|reverse> <input field> <outputfield>, servicelookup <input field> <output field> <optional services file path>). Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug' Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-6.3.0-aa7d4b1ccb80-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done [ OK ] 3) To resolve the above issue before deploying app - fixed the $SPLUNK_HOME_DEPLOYER/etc/apps/dnslookup/default/apps.conf and comment out the two lines shows below in bold-text. (i.e attribute author and description under stanza [install]) -----------$SPLUNK_HOME_DEPLOYER/etc/apps/dnslookup/default/apps.conf--------- [package] id = dnslookup [install] is_configured = 0 **#author = Travis Freeland description = dnslookup , servicelookup ** build = 1 install_source_checksum = 5224a17b098abd2d668cb74836b9c1b394c4f373 [ui] is_visible = 0 label = dnslookup [launcher] author = Travis Freeland description = dnslookup , servicelookup version = 0.1 NOTE: Since this is not Splunk provided app , but has been devloped by Travis Freeland I have provided feedback for above issue to be resolved.. 4) Once the above issue was fixed copied the app from $SPLUNk_HOME_DEPLOYER/etc/apps/dnslookup to $SPLUNk_HOME_DEPLOYER/etc/shcluster/apps/dnslookup and deployed the bundle using command $SPLUNK_HOME_DEPLOYER/bin/splunk apply shcluster-bundle -target https://:8089 Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y Your session is invalid. Please login. Splunk username: admin Password: Bundle has been pushed successfully to all the cluster members 5) The App got deployed without any issue. 6) In your case have you been able to deploy other apps besides this app. ... View more

Here is some information on this error message in summary, Configuration replication within the SHC is slower than the ideal standards (of 1 second), possibly because the SHC needs to frequently replicate the config updates with large lookup files. We currently don't support blacklisting of the lookup files in the SHC.If you want to confirm this, on each of the Search Heads, please take a backup of the latest bundle file under var/run/splunk/snapshot to a temporary directory, and make a note of their sizes. untar the bundle file (tar xvf ...) and make a note of the largest files in the bundle directory. The expectation is the largest files in the bundle would be the lookup csv files, which will confirm this. ... View more

Please note currently multi-site buckets are bonded (has persistent binding) to Originating Site. As a result once site is created it cannot be decommissioned or else the buckets Originating from that site will never meet the Search and Replication Factor. Splunk has pending Enhancement Bug requesting that bonding should be broken and it should be more flexible to copy/move bucket between sites. For your reference enhancement Request is --SPL-110192:Multi-site buckets should not be bonded to Originating Site. ... View more