HTTP 503 status code from HEC during high load

[keio_splunk]

Rsyslogd server is setup to send syslog messages to Splunk HTTP Event Collector (HEC) using omhttp module. During high load, HEC sending the following response to rsyslogd server: http status code: 503 reply: {"text":"Server is busy", "code":9, "invalid-event-number":0}. How to address this issue?

[keio_splunk · Accepted solution]

HEC input queue is full and is telling Rsyslogd server that it is busy.

The HEC queueSize has to be increased to cope with the syslog messages:
1. Set the queueSize under the [http://] stanzas in $SPLUNK_HOME/etc/system/local/input.conf. Note: indicates the token name as assigned by the user, apply to individual tokens. Settings specified here override settings specified within the [http] stanza.
2. Restart the Splunk server.

Refer to http://dev.splunk.com/view/event-collector/SP-CAAAE6Q.