The indexes.conf file applies to an individual indexer or peer node, it has no knowledge of how many other members are in the cluster or how much data might be on each member.
Therefore your indexes.conf needs to be designed per indexer.
path = /logs
maxVolumeDataSizeMB = 700000
That is 683GB per indexer (700000/1024), not 683GB of hot data for the entire cluster.
You could narrow down your dbinspect to a single indexer, dividing by the number of indexers should roughly work however that is assuming the data is evenly balanced between your indexer cluster members.
Newer splunk versions allow data rebalancing to assist with this.
I use the query:
| tstats count WHERE index="*" by splunk_server _time span=10m | timechart span=10m sum(count) by splunk_server
I then visualize in an area graph with 100% stacked mode to see if the data is even among cluster members or not.
If it's not even then you might need to do some tweaking and also run a data rebalance.
... View more
Another possible solution to this issue:
After upgrading our Splunk deployment manager from 6.4.1 to 7.3.2 I noticed I was getting this for many, but not all, of our forwarders.
All the network connectivity checks seemed to be okay. Nothing had been changed in the serverclass.conf file. Instead it turned out to be an SSL issues detailed in this bug: SPL-141961 "Older 6.0, 6.1, 6.2, 6.3 maintenance release versions unable to connect to 6.6.x and later via management port"
The true fix is to upgrade all the forwarders. But the quick solution is to change the SSL config on the deployment server in the server.conf file by adding the following to the 'sslConfig' stanza:
sslVersions = *,-ssl2
sslVersionsForClient = *,-ssl2
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
Then restart the deployment server and you should find your missing forwarders are now able to talk to it again.
A bit more detail on this can be found here on the Splunk website: https://docs.splunk.com/Documentation/Splunk/7.2.3/ReleaseNotes/KnownIssues (do a search for the issue number: SPL-141961)
... View more