Seeing ERROR message "may have returned partial results" from few indexers".  Logs from those indexers are showing following error messages.   WARN CacheManagerHandler - Localization failure has been reported, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", sid="remote_*_1646235912.747477_6AFBB424-8451-40A4-A05C-A0337BDBC296", errorMessage='waitFor probe, cache_id="bid|index_name~8264~6A0ED00A-E4AB-4B46-9F69-CD517B4C8965|", did not localize all files before reaching download_status=idle files={"file_types":["tsidx","bloomfilter","deletes"]} local_files={"file_types":["dma_metadata","strings_data","sourcetypes_data","sources_data","hosts_data","lex","tsidx","bloomfilter","journal_gz","other"]} failure_code=0 failure_reason='   Any idea, whats causing this and how to fix it? ... View more

So, 2 of our indexers sometime have very high I/O due to a known issue,  but this is causing index queueing on all our 31 other indexers in same cluster.  When we turn off the 2 indexers that are going to have high I/O,  we dont see any issue. We are assuming that replication to these problem indexers are blocking other indexers and causing ripple effect across cluster.  Is it expected that the cluster behaves this way?   Are there any configuration to optimize or create dedicated threads only for replication with out blocking indexing and searching? ... View more

Getting this error when using batch_save to update kvstore through Splunk python SDK. This works fine when posting same content through curl, not sure if some limitation through splunk SDK. Anyone have any idea on quick fix? File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/client.py", line 3724, in batch_save return json.loads(self._post('batch_save', headers=KVStoreCollectionData.JSON_HEADER, body=data).body.read().decode('utf-8')) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/client.py", line 3615, in _post return self.service.post(self.path + url, owner=self.owner, app=self.app, sharing=self.sharing, **kwargs) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/binding.py", line 289, in wrapper return request_fun(self, *args, **kwargs) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/binding.py", line 71, in new_f val = f(*args, **kwargs) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/binding.py", line 742, in post response = self.http.post(path, all_headers, **query) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/binding.py", line 1208, in post return self.request(url, message) File "/opt/splunk/etc/apps/adp_cmdb/bin/splunklib/binding.py", line 1228, in request raise HTTPError(response) splunklib.binding.HTTPError: HTTP 400 Bad Request -- The provided query was invalid. ... View more

login.splunk.com is not loading and gives error. ... View more