If this is the issue on the Search Head you need to set $SPLUNK_HOME/etc/system/local/server.conf [clustering] generation_poll_interval = 120 After making the chnage restart the Splunk service for Search head. ... View more

Could you share command used and the results you got ... View more

Starting Splunk Version 6.2 and above we have CLI command to remove Cluster Peer/Clustered Indexer . Refer ::: http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Removepeerfrommasterlist ... View more

For '|delete' to work in Splunk Indexed Clustered environment it is required that management for is open between on Cluster peers across sites. ... View more

Try these steps as it may help resolve this issue ... Shutdown the peer Deleting the slave-apps and slave-apps.old directories, and bundle under var/run/splunk/cluster/remote-bundles Sometimes, stopping CP does not stop child processes and keep updating the bundles and splunkd.log. When running into this state, you need to kill splunkd processes by (killall splunk) Create Another bundle on Cluster Master restart Cluster Peer. ... View more

Could you please change all the SHC members to have $SPLUNk_HOME/etc/system/local/server.conf scheduling_heuristic = round_robin * This setting configures the job distribution heuristic on the captain. * There are currently two supported strategies: 'round_robin' or 'scheduler_load_based'. * Default is 'scheduler_load_based' We have seen that using 'round_robin' load distribution is even between SHC members. ... View more

To get the list of excess replicated Buckets you can run the following search on the Cluster Master |rest /services/cluster/master/buckets filter=replication_count> To get the list of excess search buckets you can run the following search on the Cluster Master |rest /services/cluster/master/buckets filter=search_count> ... View more

You are correct- delete wil cause it to lose data. Log a Splunk Support Case. ... View more

There are two scenarios this could happen.. If you find the "/services/shcluster/member/artifacts//discard endpoint failing due to network/REST issues on splunkd_access.log on the members this can happen. Other possibility is lot of messages between captain/members leading to delay in messages.. For the 1st case, recommendation is to transfer captaincy to a different node.. (6.2.3 does not have transfer captaincy command) So manually transfer captaincy by bringing down the current captain will fix the current flow of WARN messages if you don't find any network issues/REST call failures it could be the delay in jobs and increasing the "executor_workers = 20" (default 10) in stanza shclustering in server.conf might help.. ... View more

1)In Splunk Version 6.2.3 In the Search Head modified the below mentioned conf files cat web.conf [settings] enableSplunkWebSSL = 1 cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK ecdhCurveName = secp384r1 sslVersions = tls1.2 allowSslRenegotiation = false cat server.conf [sslConfig] sslKeysfilePassword = $1$IzgaP3G/xTrd cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK ecdhCurveName = secp384r1 .... Please Note: If we don’t specify cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK in server.conf but add only ecdhCurveName = secp384r1 in server.conf then it shows the following: $SPLUNK_HOME/bin/splunk help Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure So in order to make everything work successfully, it needs both cipherSuite and ecdhCurveName. This happens only in Splunk 6.2.3 (build 264376) 2) In this distributed Search Environment, we have 1 SH and 2 Search-Peers(Indexers). I have added cipherSuite and ecdhCurveName inside both web.conf and server.conf for one peer [Here:10.222.30.238 but the other peer has No change] I noticed the peer which has the cipherSuite and ecdhCurveName, is able to send data successfully, that we can search on SH. But, the peer which did not had cipherSuite and ecdhCurveName. Based on the observation - all the members in the distributed environment [Here 1 SH, 2 peers] needs to have the same configuration changes in both server.conf and web.conf. 3)Next Migrated to Splunk Version 6.3.3 . f we only give ecdhCurveName and miss out cipherSuite , it works successfully. The peer is able to send the data, that can be searched from SH. ... View more

1) Bug SPL-111603:[Regression] No warn message for those skipped saved searches if hit the system-wide quota in SHC This Bug is marked as fixed in 6.4. In case you will like to try and reproduce you can take following steps 1.1. Create a SHC with 3 members 1.2. Set search quota. $splunk_home/etc/system/local/limits.conf [search] base_max_searches=1 max_searches_per_cpu=0 1.3. Create 5 scheduled real time search every minutes. curl -k -u admin:changed https://localhost:8089/servicesNS/admin/search/saved/searches -d name=testRT_1 -d search="index=_internal | head 30" -d is_scheduled=1 -d dispatch.earliest_time="rt-20m" -d dispatch.latest_time="rtnow" -d cron_schedule="* * * * " ; curl -k -u admin:changed https://localhost:8089/servicesNS/admin/search/saved/searches -d name=testRT_2 -d search="index=_internal | head 30" -d is_scheduled=1 -d dispatch.earliest_time="rt-20m" -d dispatch.latest_time="rtnow" -d cron_schedule=" * * * " ; curl -k -u admin:changed https://localhost:8089/servicesNS/admin/search/saved/searches -d name=testRT_3 -d search="index=_internal | head 30" -d is_scheduled=1 -d dispatch.earliest_time="rt-20m" -d dispatch.latest_time="rtnow" -d cron_schedule=" * * * " ; curl -k -u admin:changed https://localhost:8089/servicesNS/admin/search/saved/searches -d name=testRT_4 -d search="index=_internal | head 30" -d is_scheduled=1 -d dispatch.earliest_time="rt-20m" -d dispatch.latest_time="rtnow" -d cron_schedule=" * * * " ; curl -k -u admin:changed https://localhost:8089/servicesNS/admin/search/saved/searches -d name=testRT_5 -d search="index=_internal | head 30" -d is_scheduled=1 -d dispatch.earliest_time="rt-20m" -d dispatch.latest_time="rtnow" -d cron_schedule=" * * * *" ; 1.4)wait 2 minutes. 1.5)Check from each search heads: grep "system-wide" ~/splunk/var/log/splunk/splunkd.log -21-2015 10:09:05.896 +0800 WARN SHPMaster - Search not executed: The maximum number of real-time concurrent system-wide searches has been reached. current=1 maximum=1 for search: admin;search;testRT_1 12-21-2015 10:09:10.952 +0800 WARN SHPMaster - Search not executed: The maximum number of real-time concurrent system-wide searches has been reached. current=1 maximum=1 for search: admin;search;testRT_2 2> SPL-111347:SHC: Search and scheduler performance degrade with increasing number of saved searcher This performance degradation was seen only when the saved searches were in thousands. ... View more

This error is for hot buckets that are being index on the Originating node - but the issue is Data is not being replicated to any other Node to meet the required Replication and Search Factor. This Data will still be searchable from the Originating node- but if the Originating Nodes goes down the data will not be available on any other node. These error messages will disappears as these these buckets will roll from hot to warm. Upon roll of bucket Cluster Master will check if the RF and SF are not met - it will force copy of bucket to meet RF and SF. It may take days for bucket to roll - In case you need to address this issue right away you have few options. 1) Easy approach will be to issue rolling restart from Cluster the hot buckets will roll to warm and the issue wil lget addressed. or 2) Other option will be to look for index have buckets in-such state and issue the command that will roll the hot bucket .Ran the following curl command on internal index _audit and roll the buckets from hot to warm curl -k -u admin:changeme https://HOST:PORT/services/data/indexes/YOUR_INDEX/roll-hot-buckets -X POST HOST is the StreamingSource server name, PORT being 8089, YOUR_INDEX is the index needing to roll example of _audit curl -k -u admin:PASSWORD https://idx2.coxauto.splunkcloud.com:8089/services/data/indexes/_audit/roll-hot-buckets -X POST You will need to run this for each of teh index with buckets in run on remaining indexes found here: ... View more