We have a Splunk cluster that is shared by multiple users/teams. We've set up an API user that makes calls to Splunk at a constant rate. Most of the time everything works fine, but every once in a while we see queries getting queued (for a short duration, but that is unacceptable for our specific use case).
We tried increasing the relevant configs in limits.conf, but the issue keeps occurring. Our assumption is that the API is getting queued when there are lots of other queries (from other users) running.
How can we guarantee that queries by the API user don't get queued? Is there a way to give a specific user/role a dedicated quota? Is there a way to enforce a limit on the quota of all other users/roles?
