I wrote a simple, REST-based proxy to query Splunk's REST API from SiteScope. The proxy manages job creation, tracking etc so that SiteScope can simply issue a GET on a URL and get easily parsable XML in return. I have 25 or so monitors that use it, and they run once per minute.
Very often the proxy gets a 503 server unavailable message from Splunk. Apparently I'm hitting a limit on the API interface? Is there a switch to adjust this?
v4.1.5
EDIT2:
My $SPLUNK_HOME/etc/system/local/authorize.conf file:
[default]
run_web_script_fields = enabled
run_web_script_surrounding_events = enabled
[role_user]
srchJobsQuota = 16
The user I am hitting the API with is in the "user" role.
Try this in server.conf:
[managementServer]
maxBackLog = 100
requestQueueSize = 100
threadPoolSize = 100
More research from the SiteScope end: When I get the error there are at most 2 or 3 other searches running. Doesn't seem like too many.
Nope. Still getting the 503s. 😞
Found this previous question:
Seems to be the answer