Why am I receiving a "Server is busy" error after configuring the HTTP Event Collector?

[ecoquelin]

Dear all,

I have configured the HTTP Event Collector but can't successfully send events.

My configuration in inputs.conf

[http]
allowSslCompression = true
allowSslRenegotiation = true
dedicatedIoThreads = 2
disabled = 0
enableSSL = 0
index = ffjj
maxSockets = 0
maxThreads = 0
sslVersions = *,-ssl2
_rcvbuf = 1572864
host = splunk-dev
port = 8088
sourcetype = R_LICENCIE_TEMP
useDeploymentServer = 1

[http://appmobile]
disabled = 0
host = splunk-dev
index = appmobile
indexes = appmobile
sourcetype = _json
token = 03F50C74-121B-4FBF-9999-ACB9A032AD02
sourcetypeSelection = From List

I have created a very basic request

{
    "time": 1433188255, 
    "event": {
        "membre_no" : 1213,
        "est_membre": 1
    }
}

I know Splunk receives the message but it throws an error 503 "Server is busy"

{
"text": "Server is busy"
"code": 9
}

my request is being sent to http://:/services/collector/event

I have deactivated SSL in the HTTP Event Collector configuration. I know it is taken into account because if activated, there server doesn't reply.

I would like to investigate but :

1. I can't find anyone having the same issue as me - no topic relates to 503 - "server is busy"
2. I don't know how to increase log level for HTTP Event collector. Setting this category category.HttpEventCollector=DEBUG doesn't provide more logs (and I update the rootCategory level as well)...
3. I know the parsing is being performed by Splunk because as soon as I change the JSON format to something malformed, I get another error

Can you please let me know what's going on and how I can have logs?

Thank you in advance for your help.

Eric

[alexgohberg]

Hey i solved it by disabling the Use Deployment Server checkbox under global settings in HTTP Event Collector.

[Huu-Nguyen]

I gotta login to say thank you. You have saved me hours of fixing.

[0YAoNnmRmKDg]

this just took me 2 hours to resolve! thank you for posting back - what an odd behavior!

[MVREID]

Problem solved, was due to http collector being configured on heavy forwarder and not from the deployment server.

[starcher]

yeah never send useDeploymentServer = 1 in the config you push to the HEC receiver. you want that setting only on at the DS itself. It tells Splunk to look for the HEC config in $SPLUNK_HOME/etc/deployment-apps folder. Older versions ignored it. Somewhere around 6.4 the behavior changed.

[MVREID]

Don't have an answer, but curious if you ever resolved. I have the same issue in a distributed deployment.
thanks

[ecoquelin]

In addition, I found that in the log file after having started splunk with --debug

09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - Before accept
09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - Creating polled fd from factory
09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - adding connection to factory created fd = 0x7f904f02e000
09-21-2016 21:29:40.627 +0000 INFO  TcpChannel - Accepted connection
09-21-2016 21:29:40.633 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::http:appmobile|host::mydomain:8088|_json|
09-21-2016 21:29:40.633 +0000 DEBUG PropertiesMapConfig - Pattern '_json' matches with priority 100
09-21-2016 21:29:40.633 +0000 DEBUG HttpInputDataHandler - handled token: 03F50C74-121B-9999-AA2C-ACB9A032AD02 channel: n/a reply: 9 processed 1