Testing the HTTP Event Collector, why am I getting a "Server is busy" message?

[brent_weaver]

 curl -k  https://localhost:8088/services/collector/event -H "Authorization: Splunk 8F6CCFXA-6D7B-48BE-A59F-7361D6003422" -d '{"event": "hello world"}'
{"text":"Server is busy","code":9}

Any help is much appreciated. Thanks!

[stevenchamales]

BLUF: Clearing my indexer acknowledgement cache fixed this problem for me.

I ran into this problem while running a Python script I wrote that extracted log data from a PostgreSQL AWS RDS and forwarded it into a Managed Splunk Cloud instance. The script ran fine for a while, then out of the blue new events no longer showed up in Splunk. The Splunk index my events were being sent to was not full, and events from other sources in our architecture continued to index as they always had.

I ended up writing a python script that cleared out the "acks" for the channel UUID I was using, and then my RDS-to-HEC forwarder ran fine after that.

I don't have permissions to access the other HEC input tokens and the indexes the indexes they're used to send data to, so I don't know if this was an issue of too many acks still pending query across the entire system, or if my HEC token/Channel combo specifically had too many. Regardless, I'm fairly certain that the buildup of un-queried indexer acknowledgements had something to do with my forwarder failing because of {"text":"Server is busy","code":9} errors.

See docs about Indexer Acknowledgement:
http://dev.splunk.com/view/event-collector/SP-CAAAE8X

My python script:

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
import json
import sys
from ast import literal_eval

requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
h = {}
h['X-Splunk-Request-Channel'] = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
h['Authorization'] = 'Splunk xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
u = 'https://input-prd-p-xxxxxxxxxx.cloud.splunk.com:8088/services/collector/ack'
r = {}
r['acks'] = {'0':'false'}
i = 0
n_trues = 0
while i < 1000000000:
p = {"acks":[i,i+1,i+2,i+3,i+4,i+5,i+6,i+7,i+8,i+9]}
r = requests.post(u, json=p, headers=h, verify=False).json()
i += 10
l = [k for k in r['acks'].keys() if r['acks'][k] == True]
n_trues += len(l)
if os.path.isfile("stop.txt"):
print "quitting."
exit()
with open("progress.txt",'w') as f:
s = str(i) + " , " + str(n_trues)
f.write(s)
f.close()
sys.stdout.write('.')
sys.stdout.flush()
with open("progress.txt",'a') as f:
f.write("\n")
json.dump(r,f)

[hunters_splunk]

Hi Brent,

Maybe you can try this:

curl -k  "https://localhost:8088/services/collector"
 -H "Authorization: Splunk 8F6CCFXA-6D7B-48BE-A59F-7361D6003422" 
 -d '{"event": "hello world"}'

Thanks!