dont post wrong answers to delete all the data ... View more

For the manual localize endpoint mentioned: curl -ku admin:changeme "https://localhost:8089/services/admin/cacheman/_localize" -X POST  Will it be localizing of all the buckets, of all the smartstore enabled indexes?  What are the specific endpoints for localizing an individual bucket or index? ... View more

Anyone else facing same issues in 8.2.4. Will check with support and see. ... View more

Just to update this thread and share that this stills working My context: I deployed recently a cluster env. and did not noticed that i my filesystem was getting close to the limit I realized when all my indexers status=AutomaticDetention As i was aware that the data was useless i followed your approach and worked ... View more

Tried in Splunk 8.1.3 Removed the   ../<indexname>/db/<bucket_id>/rawdata/deletes folder. Didn't work.  ... View more

Oustanding solution. We ended up using it for deleting buckets that were causing issues in our cluster. ... View more

Faced exactly the same issue. Issue resolved after adding "repFactor = auto" in test though in production everything is working fine without "repFactor = auto".   ... View more

how can you fix this issue when you have thousands of buckets in pending? I have around 4800 buckets and i cannot go and execute roll command on each and every Index right? Can you give me any solution for this please? Thanks for the answer.   @jbarlow_splunk  ... View more

omg that is exactly the problem!! I cant believe its so dumb lol.. the second question I guess would be how can we change the mgmt port from 8089, like if i wanna use 443 only for communication with the indexer is that possible? I will accept this as the answer btw, this is definitely the solution. ... View more

this shouldve been something we fixed/alleviated in 7.3.2. most likely these were unclean bucket replications - where a StreamingError happened on a hot bucket target, which made the indexers throw one copy away, and triggered a fixup to re-replicate the missing bucket... one thing to try to pinpoint this is to pick any bucket that needed fixups, and see what happened to that bucket throughout the process. ... View more

please see https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/Migratetomultisite#Post-migration_bucket_behavior for more details. if we want multisite capabilities for our old buckets, we can toggle the appropriate flag (and never rely on replication_factor ) ... View more

splunk_server currentDBSizeGB maxTotalDataSizeGB totalEventCount eai:acl.app 1 splunk-idx11 384 500 2571516578 _cluster 2 splunk-idx12 272 500 1827380397 _cluster 3 splunk-idx13 419 500 2893046932 _cluster 4 splunk-idx20 68 500 521531899 _cluster 5 splunk-idx21 72 500 566775305 _cluster 6 splunk-idx22 71 500 552783442 _cluster 7 splunk-idx23 77 500 611006777 _cluster 8 splunk-idx24 77 500 602144046 _cluster 9 splunk-idx25 67 500 524328395 _cluster 10 splunk-idx26 70 500 549058168 _cluster 11 splunk-idx27 75 500 589072469 _cluster 12 splunk-idx28 63 500 488439975 _cluster 13 splunk-idx29 67 500 516032474 _cluster ... View more

when searched for data out of (hotlist_recency_secs period ) the indexer will get the required warm/cold buckets from remote storage and stores in local cache , so it is recommended to have SSD's for your local storage for better cache management and performance. ... View more

do u want 1 copy per site? origin:1,total:2 this puts 1 per site. or equivalently: origin:1,site1:1,site2:1,total:2 for SF, if you don't want both copies to be searchable, you can do: origin:1,total:1 (only the source copy will be searchable) ... View more

Now its replicating. Thanks. ... View more

Sadly had similar issues with even 7.1.1. I suspect that its related to Data Models as one of our indexers that doesn't deal with any data models etc was fine, were as the rest of our Indexers CPU, Memory disk and disk I/O went insane to the point where they would eventually crash. Rolled back to 7.0.2 for now. Opened case as well. ... View more

Hi thank you for the answer. I can´t answer the question, the file seems to be temporay, and its deleted pretty fast. But I do see folders named like "DM_Splunk_SA_CIM_Authentication" with files like : done metadata_checksum metadata.csv What can I do with these errors since there floding my splunkd.log I would like to get rid of them? ... View more

Hi, @harsmarvania57, i have a similar requirement. Can you provide me the steps how to migrate the date from one multi-site index cluster environment to an another multi-site environment? I have 2 sites, each site has 3 peers (total six peers) in one environment and the same configuration in an another environment. Should I copy buckets from all the indexers in the old environment to all the indexers in the new environment? ... View more

I've attempted to provide an exhaustive answer to this at What is the best practice to address the "..is neither in the bundle downloaded from master nor managed by local deployment client..." validation failure? ... View more

I have actually. So from the CM i am resolve the ip of the Indexer and vice versa: Note i did notice the "null" problem but its not really a problem it still resolves and i looked it up its for alot of reasons.. CM: ` nslookup w.x.y.z nslookup: can't resolve '(null)' Name: w.x.y.z Address 1: w.x.y.z ip-w-x-y-z.ca-central-1.compute.internal Indexer: nslookup a.b.c.d a.b.c.d.in-addr.arpa name = ip-a-b-c-d.ca-central-1.compute.internal ` I have resorted to pointing directly via IP now because at this point i am testing out register_replication_address. I have tested things like tcpdump traffic from the CM to the indexer and I am not even seeing the CM reach out to the indexer on that IP..So confused I have a question here if you want more details: https://answers.splunk.com/answers/818038/cluster-master-register-replication-address-invali.html ... View more

Yes, summary replication got back up again. How are things with your issue ? Were the support helpful ? @dxu_splunk ... View more

See SPL-135257 @ http://docs.splunk.com/Documentation/Splunk/6.5.1/ReleaseNotes/Knownissues#Indexer_and_indexer_clustering_issues hidden files can cause bundle checksum mismatches between versions. either dont push a bundle with hidden files while in mixed-version clustering (for some specific versions), or don't put hidden files into a bundle! ... View more

See the ldap answer above ... View more

We are using splunk version 6.4.0. I am aware that replication and search factor entries are present in default/server.conf too but there might be a case where our customer would have set custom replication and search factors in his local/server.conf file. ... View more

I have same issue my data is getting doubled in indexers each time a log is captured ... View more