I was wondering if there was a recommended way to ensure that an app doesn't stream too high a volume of data to avoid flooding the import queue.
For example, when running my app, I may receive messages such as the following in metrics.log:
01-10-2017 03:17:01.059 -0800 INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=803, largest_size=831, smallest_size=0
I'm aware that I can increase the limits within server.conf, but I'd prefer to throttle the app in case it's moved onto another instance.
Also, when a queue is blocked or the system is otherwise flooded with events, are they always cached until the queue is free or else are they lost each time?
Thanks for input on either of these.
In answer to your first question, I don't think you can adjust the queue size for just one app - server.conf settings are global system settings.
As to your second question:
Queue is bounded by memory and gets blocked if it is full. Persistent Queue does not get blocked because it uses the file system
(like a virtual memory). Therefore, you can use persistent queues to help prevent data loss.
To create the persistent queue, specify these two attributes within the particular input's stanza:
persistentQueueSize = <integer>(KB|MB|GB|TB) * Max size of the persistent queue file on disk. * Defaults to 0 (no persistent queue).
Hope this helps. Thanks!
I'm aware that servers.conf is global but thank you nonetheless for clarifying that.
I believe persistent queues are exactly what I'm looking for. Thank you very much for this information.