How to avoid overloading queues when using XML streaming?

[Vidd]

Hi,

I was wondering if there was a recommended way to ensure that an app doesn't stream too high a volume of data to avoid flooding the import queue.

For example, when running my app, I may receive messages such as the following in metrics.log:

01-10-2017 03:17:01.059 -0800 INFO  Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=803, largest_size=831, smallest_size=0

I'm aware that I can increase the limits within server.conf, but I'd prefer to throttle the app in case it's moved onto another instance.

Also, when a queue is blocked or the system is otherwise flooded with events, are they always cached until the queue is free or else are they lost each time?

Thanks for input on either of these.

[hunters_splunk · Accepted solution]

Hi Vidd,

In answer to your first question, I don't think you can adjust the queue size for just one app - server.conf settings are global system settings.

As to your second question:
Queue is bounded by memory and gets blocked if it is full. Persistent Queue does not get blocked because it uses the file system
(like a virtual memory). Therefore, you can use persistent queues to help prevent data loss.
To create the persistent queue, specify these two attributes within the particular input's stanza:

persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk.
* Defaults to 0 (no persistent queue).

For example:

[tcp://9994]
persistentQueueSize=100MB

For details, see http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Usepersistentqueues.

Hope this helps. Thanks!
Hunter

[Vidd]

Hello, hunters.

I'm aware that servers.conf is global but thank you nonetheless for clarifying that.

I believe persistent queues are exactly what I'm looking for. Thank you very much for this information.