I am experiencing an unresponsive splunkd with an indexer instance and a deployment server instance on the same box. It was working well at 700-800 forwarders, but when we went to about 1000 it stopped working. All forwarders are connecting to both the DS instance and the indexer to send data over SSL.
By unresponsive, I mean splunkd will respond to requests (from clients, from ./splunk list deploy clients, from ./splunk login) for a few seconds every couple of minutes.
Hardware is 16x2.4Ghz cores, 48GB memory, running Suse Linux 11.0.
... View more