Getting Data In

How to avoid overloading queues when using XML streaming?

Vidd
Explorer

Hi,

I was wondering if there was a recommended way to ensure that an app doesn't stream too high a volume of data to avoid flooding the import queue.

For example, when running my app, I may receive messages such as the following in metrics.log:

01-10-2017 03:17:01.059 -0800 INFO  Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=803, largest_size=831, smallest_size=0

I'm aware that I can increase the limits within server.conf, but I'd prefer to throttle the app in case it's moved onto another instance.

Also, when a queue is blocked or the system is otherwise flooded with events, are they always cached until the queue is free or else are they lost each time?

Thanks for input on either of these.

0 Karma
1 Solution

hunters_splunk
Splunk Employee
Splunk Employee

Hi Vidd,

In answer to your first question, I don't think you can adjust the queue size for just one app - server.conf settings are global system settings.

As to your second question:
Queue is bounded by memory and gets blocked if it is full. Persistent Queue does not get blocked because it uses the file system
(like a virtual memory). Therefore, you can use persistent queues to help prevent data loss.
To create the persistent queue, specify these two attributes within the particular input's stanza:

persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk.
* Defaults to 0 (no persistent queue).

For example:

[tcp://9994]
persistentQueueSize=100MB

For details, see http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Usepersistentqueues.

Hope this helps. Thanks!
Hunter

View solution in original post

hunters_splunk
Splunk Employee
Splunk Employee

Hi Vidd,

In answer to your first question, I don't think you can adjust the queue size for just one app - server.conf settings are global system settings.

As to your second question:
Queue is bounded by memory and gets blocked if it is full. Persistent Queue does not get blocked because it uses the file system
(like a virtual memory). Therefore, you can use persistent queues to help prevent data loss.
To create the persistent queue, specify these two attributes within the particular input's stanza:

persistentQueueSize = <integer>(KB|MB|GB|TB)
* Max size of the persistent queue file on disk.
* Defaults to 0 (no persistent queue).

For example:

[tcp://9994]
persistentQueueSize=100MB

For details, see http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Usepersistentqueues.

Hope this helps. Thanks!
Hunter

Vidd
Explorer

Hello, hunters.

I'm aware that servers.conf is global but thank you nonetheless for clarifying that.

I believe persistent queues are exactly what I'm looking for. Thank you very much for this information.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...