It is possible that there are other things going on that is causing this error than what is stated above.  Since I identified a unique root cause I wanted to share with all.  The last bullet below was what worked for me but the below bullets represents a summary of recommended steps to get to root cause for this. First verify that the size of the bundle being sent from SH is not greater than the bundle size limit setting on the SH (maxBundleSize distSearch.conf) or the Indexer (max_content_lengh server.conf) Then check for permissions/ownership errors on all the instanced by running “ls -lahR /opt/spunk | grep root” Then run ./splunk btool check Then check the CM bundle details and compare if the latest active bundle in the peers is same as the CM. Then run the top command to see if there are any resources using a significant percentage of CPU utilization over Splunk.  A new application could have been introduced that is preventing writes from taking place over a long period of time due to files being locked by other application.  This can be further verified by: Run the following on each indexer “sudo tcpdump <ipaddressofsourceSH>” then attempt to run your search from the SH and see if you see the commands coming over. If fails that there is an application that on in your environment that is preventing Splunk from doing what it need to do and you need to apply for an Splunk exceptions for the recently introduced application. ... View more

This happens when you keep cluster master in maintenance mode and re-add peer to cluster. Always ensure you keep cluster master out of maintenance mode when you re-add peer. You can simply fix this by going to splunkweb on Search head _ settings_distributed Search_ search peers . Select the peer which is having issues and add the admin user/password _ save ... View more

if you're seeing the corrupt only on hot buckets, you need to change the SPL to retrieve. |dbinspect index=* corruptonly=true | search state!=hot Or |dbinspect index=_internal corruptonly=true | search NOT state=hot The reason we exclude hot buckets is because  the status of those hot buckets are in transient and they are still being updated(written). ... View more

try this: index=operatingsys host=hf1 OR host=hf2 or host=hf3 source="/var/run/syslog/*" | rex .......... | table host listenername PID | append [|inputlookup CSV_LIST | fields host listenername ] | stats latest(*) as * by host listenername By the way, if your CSV has different field names in it, you can use an eval or rename in the subsearch. All the best ... View more

Your ulimits are not set correctly, or are using the system defaults. As a result, splunkd is likely using more memory than allowed or available, so the kernel kills the process in order to protect itself. ... View more