I have this question as a reference: Splunk Question
I have one indexer, SH, and one forwarder. At some point, I had sent data from the Forwarder to the indexer and it was searchable from the SH. After a few runs, I had received this error:
Search not executed: The minimum free disk space (5000MB) reached for /export/opt/splunk/var/run/splunk/dispatch. user=admin., concurrency_category="historical", concurrency_context="user_instance-wide", current_concurrency=0, concurrency_limit=5000
This was because our root filesystem was only 20 gigs and after a few searches it had dropped below 5 gigs. So at this point i was wanting to remount and move around a few things and move splunk into a larger FS. So i did, and since then, I have fully removed and reinstalled splunk and remounted it multiple times on each server, and some of these issues persist. Currently, The search Head isnt wanting to connect and search the indexer. Storage and server resources are fine, and i can connect through telnet over port 8089, but the replication keeps failing.
I keep receiving error:
Bundle replication to peer named dc1nix2dxxx at uri https://dc1nix2dxxx:8089 was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
Expected common latest bundle version on all peers after sync replication, found none. Reverting to old behavior - using most recent bundles on all
I can connect and do everything else on the server through the backend, and i can telnet between the two, so im not sure what to do. Most everything i keep seeing has me check settings under distributed environment. And most of the time, under those setting, it says due to our license we arent allowed to access those settings. All i have set is one search peer, dc1nix2pxxx:8089.
Some sources say its an issue with the web.conf settings, but i dont have a web.conf under my local, and if i did, what should that look like? I just have three servers im working with.
Id appreciate any help or guidance in the right direction.
Thank you
