Getting Data In

After configuring the HTTP Event Collector, why am I receiving a "Server is busy" error?

Explorer

Dear all,

I have configured the HTTP Event Collector but can't successfully send events.

My configuration in inputs.conf

[http]
allowSslCompression = true
allowSslRenegotiation = true
dedicatedIoThreads = 2
disabled = 0
enableSSL = 0
index = ffjj
maxSockets = 0
maxThreads = 0
sslVersions = *,-ssl2
_rcvbuf = 1572864
host = splunk-dev
port = 8088
sourcetype = R_LICENCIE_TEMP
useDeploymentServer = 1

[http://appmobile]
disabled = 0
host = splunk-dev
index = appmobile
indexes = appmobile
sourcetype = _json
token = 03F50C74-121B-4FBF-9999-ACB9A032AD02
sourcetypeSelection = From List

I have created a very basic request

{
    "time": 1433188255, 
    "event": {
        "membre_no" : 1213,
        "est_membre": 1
    }
}

I know Splunk receives the message but it throws an error 503 "Server is busy"

{
"text": "Server is busy"
"code": 9
}

my request is being sent to http://:/services/collector/event

I have deactivated SSL in the HTTP Event Collector configuration. I know it is taken into account because if activated, there server doesn't reply.

I would like to investigate but :

  1. I can't find anyone having the same issue as me - no topic relates to 503 - "server is busy"
  2. I don't know how to increase log level for HTTP Event collector. Setting this category category.HttpEventCollector=DEBUG doesn't provide more logs (and I update the rootCategory level as well)...
  3. I know the parsing is being performed by Splunk because as soon as I change the JSON format to something malformed, I get another error

Can you please let me know what's going on and how I can have logs?

Thank you in advance for your help.

Eric

Explorer

Hey i solved it by disabling the Use Deployment Server checkbox under global settings in HTTP Event Collector.

Path Finder

this just took me 2 hours to resolve! thank you for posting back - what an odd behavior!

0 Karma

Path Finder

Problem solved, was due to http collector being configured on heavy forwarder and not from the deployment server.

0 Karma

SplunkTrust
SplunkTrust

yeah never send useDeploymentServer = 1 in the config you push to the HEC receiver. you want that setting only on at the DS itself. It tells Splunk to look for the HEC config in $SPLUNK_HOME/etc/deployment-apps folder. Older versions ignored it. Somewhere around 6.4 the behavior changed.

0 Karma

Path Finder

Don't have an answer, but curious if you ever resolved. I have the same issue in a distributed deployment.
thanks

0 Karma

Explorer

In addition, I found that in the log file after having started splunk with --debug

09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - Before accept
09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - Creating polled fd from factory
09-21-2016 21:29:40.627 +0000 DEBUG TcpChannel - adding connection to factory created fd = 0x7f904f02e000
09-21-2016 21:29:40.627 +0000 INFO  TcpChannel - Accepted connection
09-21-2016 21:29:40.633 +0000 DEBUG PropertiesMapConfig - Performing pattern matching for: source::http:appmobile|host::mydomain:8088|_json|
09-21-2016 21:29:40.633 +0000 DEBUG PropertiesMapConfig - Pattern '_json' matches with priority 100
09-21-2016 21:29:40.633 +0000 DEBUG HttpInputDataHandler - handled token: 03F50C74-121B-9999-AA2C-ACB9A032AD02 channel: n/a reply: 9 processed 1
0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!