I wanted to share you information on steps you can take to kill the searches that take memory beyond a threshold. Refer: http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/limitsconf and three attribute that you need to review are enable_memory_tracker = * If memory tracker is disabled, search won't be terminated even if it exceeds the memory limit. * Must be set to if you want to enable search_process_memory_usage_threshold or * search_process_memory_usage_percentage_threshold * By default false. search_process_memory_usage_threshold = * To be active, this setting requires setting: enable_memory_tracker = true * Signifies the maximum memory in MB the search process can consume in RAM. * Search processes violating the threshold will be terminated. * If the value is set to zero, then splunk search processes are allowed to grow unbounded in terms of in memory usage. * The default value is set to 4000MB or 4GB. search_process_memory_usage_percentage_threshold = * To be active, this setting requires setting: enable_memory_tracker = true * Signifies the percentage of the total memory the search process is entitled to consume. * Any time the search process violates the threshold percentage the process will be brought down. * If the value is set to zero, then splunk search processes are allowed to grow unbounded in terms of percentage memory usage. * The default value is set to 25%. * Any number set larger than 100 or less than 0 will be discarded and the default value will be used. ... View more

The workaround is to disable conditional expansion by setting following on the deployer and the Search Head. [search] enable_conditional_expansion=false ... View more

Just to Sum up : REMOVE FROM ONE Peer: curl -k -u admin:changeme "https://CM:8089/services/cluster/master/buckets/_audit~5~A5F789C3-22C0-407C-9B6B-10C8705F1C3D/remove_from_peer" -d peer=GUID_FOR_PEER REMOVE FROM ALL: curl -k -u admin:changeme -X POST "https://CM:80890/services/cluster/master/buckets/ _audit~5~A5F789C3-22C0-407C-9B6B-10C8705F1C3D/remove_all" ... View more

In non muli-site clustering, its either 0x0, or 0xFFFFFFFF , basically primary or not primary. The individual bits are only relevant in multi-site clustering. The flags is a 64 bit bitmask, with the smallest bit corresponding to Primary for site0. The second smallest would be primary for site1, the third for site2, and so on....so 0x0 = primary for nothing (searches from any site will not get results for this bucket on this peer) Bucket marked 0x1 = searches that come from searchheads with site=0 will get results! (primary for site0) Bucket marked 0x2 = primary for site1, all site=site1 searches will get results from these buckets Bucket marked 0x3 == (0x1+0x2) primary for site0 + site1, so searches from site0 SearchHeads and site1 searchheads will get results for this bucket. On the cluster/master/buckets/BID endpoint, the masks should add up to whatever the mutl-site config is for example, if We have available_sites=site2,site3, then the mask will need to have 0x1 (site0), 0x4(site2), 0x8 (site3) distributed among its indexers. in this example, if search_factor=1, then only 1 bucket will be searchable and should get assigned all the flags (0x13) ... View more

You can use curl command to delete bucket from a peer. Say you have Bucket rest endpoint showing bucket _audit~5~A5F789C3-22C0-407C-9B6B-10C8705F1C3D on two peer like below and you need to delete it from peer with GUD>9D08E9A4-25E1-41ED-876A-737F32840B83 You can use curl command like curl -k -u admin: https://CM_URL:8089/services/cluster/master/buckets/_audit~5~A5F789C3-22C0-407C-9B6B-10C8705F1C3D/remove_from_peer -X POST -d peer=9D08E9A4-25E1-41ED-876A-737F32840B83 Once the curl command is run you will see messages like below <?xml version="1.0" encoding="UTF-8"?> <!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .--> <?xml-stylesheet type="text/xml" href="/static/atom.xsl"?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"> <title>clustermasterbuckets</title> <id>https://CM:8089/services/cluster/master/buckets</id> <updated>2017-10-18T22:41:03-07:00</updated> <generator build="e21ee54bc796" version="6.6.3"/> ... View more

some further update for errors like below 08-01-2017 10:03:37.694 -0700 WARN ConfReplicationThread - Error pulling configurations from captain=https://:8089, consecutiveErrors=2 msg="Error in fetchFrom, at=ae823222d0607652969d338bb793469fb7de85cd: Network-layer error: Connect Timeout Please not that consecutiveErrors is not larger than 10 is not considered as a real issue. It can be due to the captain side is busy and not be able to response in time. Check what is the consecutiveErrors count for you using search like Index=_internal ( host= OR host= OR host= OR host=) source="splunkd.log" "ConfReplicationThread - Error pulling configurations from captain" | stats max(consecutiveErrors) by host It's not an issue is the consecutiveErrors<10. In case error is above 10 log case with Splunk Support ... View more

I have this same issue, could you clarify " It was mis-configured SSL settings.." ... View more

Set CM in Maintenance mode. stop all CP's, Stop CM. ./splunk enable maintenance-mode ./splunk stop. 2.On each CP cd into directory: ./splunk/var/lib/splunk//db 3. Run command grep -Rle '1$' --include bucket_info.csv Verifies these are the only buckets marked as frozen example: "indextime_et","indextime_lt","frozen_in_cluster" 1484368345,1484385061,1 Run command ( to unfreeze the bucket) grep -Rle '1$' --include bucket_info.csv | xargs sed -i 's/1$/0/g' Locates frozen buckets and replace the 1 (frozen=true) to 0 (frozen=false) Run same steps (2-4) for colddb directory: : ./splunk/var/lib/splunk//colddb Locate .bucketManifest file by running ls -la within ./splunk/var/lib/splunk/ directory. Move .bucketManifest file outside of splunk into tmp directory, during startup the file will regenerate with new information. Started up CM, set in Maintenance, start up CPs, take CM out of Maintenance: ./splunk disable maintenance-mode ./splunk show maintenance-mode ... View more

Thanks for sharing that max_searches_per_process=1 resolved it. ... View more

Yes, we can survive failures, i.e. we won't stop indexing, but we cannot possibly meet replication policy for those source indexers that have a now failed peer in their target list. CM recovery and fixup needed asap, in that case In other words: We really need CM HA. ... View more

correct.in which case new buckets dont meet rf/sf ... View more

each indexer remembers the LAST list of targets the CM gave it. if the CM is down, it will continuously use the same targets for new hot buckets. to be more correct, each indexer remembers the "last response of a new hot bucket request", and reuses that response ... View more

When an indexer creates a hot bucket, it follows this procedure: 1) Indexer needs to create a new hot bucket 2) Indexer asks the CM whom to replicate the new hot bucket to 3) CM receives the request, checks the configured RF/SF, and selects indexers (randomly) as "targets" that the original indexer should stream to. For example, if RF=3, SF=2, it will respond with two targets, one of which will also be searchable (to satisfy SF=2). All hot bucket replications are raw data ONLY. For the example above, the one target that is also told to be searchable will create its own tsidx files based on the rawdata that comes in. ... View more

On the cluster master you can use the search like | rest /services/cluster/searchhead/generation splunk_server=local | table is_searchable replication_factor_met, search_factor_met ... View more

1) remove excess buckets (cmtruncjob with size=0 is the same as remove a bucket) 2) mismatch size of a bucket - generally after rolling restart / shutdown of a peer where the streaming connection of a hot buckett isnt consistent across the indexers - in which case we truncate to the smallest size of a bucket. we truncate to the smallest size because we dont ACK back to the forwarder until ALL copies of a hot bucket get a slice, so the assumption is that the the smallest size of the bucket has all the ACK'd slices (whereas the larger bucket has a piece that hasnt been ACKd) ... View more

The thawed bucket does not work for replication, Cluster Master does not consider these buckets for replication, but throws series of error messages. Splunk has Bug open to suppress such error message. Current workarounds is.... 1) unthaw as a standalone bucket (follow the standalone bucket naming convention). note: make sure nobody unthaws the same bucket somewhere else - otherwise we may get dupe results. 2) don't use thawdb - unthaw it as a clustered bucket into db/ or colddb/ So Basically recommendation is to allow for thawing just to one location, and then simply not replicate that bucket? So ideally restore thawed bucket in Non cluster indexer and search it from Clustered Search head , by adding this standalone indexer( with thawed buckets ... View more

Shutdown all of the indexers and modify indexes.conf and change the retention back to higher value. Assuming mistake was only on one peer (or one site), you'll lose buckets on that peer. The other buckets will still be around, but marked as frozen. You need to take the folloing steps to unmark these buckets from “frozen” to “non-frozen” i)Get a list of frozen buckets using curl commands like https:// :8089/services/cluster/master/buckets?filter=frozen=true https://:8089/services/cluster/master/buckets?filter=frozen=true&filter=index= OR, from the CM you can run | rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen ii)Now for each of these bucket edit “bucket_info” and flip the frozen flag from 1 to “” For example for bucket $SPLUNK_HOME/var/lib/splunk//db/db_1476118503_1476117359_6_6DC24692-777A-424E-AE9D-5FD23FAC185B/ bucket_info or from the CM you can run | rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen Change from> "indextime_et","indextime_lt","frozen_in_cluster" 1475514557,"",1 change to> "indextime_et","indextime_lt","frozen_in_cluster" 1475514557,"","" Also for index, delete $SPLUNK_HOME/var/lib/splunk//db/.bucketManifest After for all the bucket_info is fixed for all index and indexer, you may restart the indexers. ... View more

Refer - https://conf.splunk.com/session/2015/conf2015_PLucas_Splunk_SplunkEntWhatsNew_MakingTheMostOf.pdf Thsi document discussed Schedule search window. Give a schedule window to searches that don’t have to run at specific time Example For a given search, it’s OK if it starts running someHme between midnight and 6am, but you don't really care when specifically A search with a window helps other searches Search windows should not be used for searches that run every minute Search windows must be less than a search’s period ... View more

This is still true. ... View more