Impact of increasing the queue size?

[chintan_shah]

I am facing issues while searching any logs and its takes a lot of time to index the log.
While investigating, i am seeing many times the queues are blocked as mentioned below.

INFO Metrics - group=queue, name=indexqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1466, largest_size=1466, smallest_size=0

Is it worth to increase the max size of the queues? Also while config file i need to change to increase the same?

[gjanders]

Increasing the queue size may work to add an additional buffer for bursts of incoming log data, however it is unlikely to fix your blocked indexing queue.

For example if you made you incoming TCP input queue bigger, then more data can queue there while the data is getting written to disk, you can do this with various other queues to allow a bigger buffer as data travels down the queue.

However the only real way to resolve an index queue issue on an indexer would be to index less (i.e. add more indexers) or to have faster I/O (get faster disk for your hot section). if your data is getting forwarded then you would look at the system getting forwarded to as well.

Good luck!

[ddrillic]

We had an interesting discussion about it recently at What's the maxSize we can set for the event-processing queues?

It's crucial to adjust the queue's sizes as you compensate for slow I/O with memory cashing - marvelous!!

Can you please post your indexer's $SPLUNK_HOME/etc/system/local/server.conf?

[harsmarvania57]

Hi @chintan_shah,

Indexqueue blocking is due to many reason.

1.) Storage latency - if indexer is not able to write in storage at required IOPS then indexqueue will be filled. -> You need to check with storage team whether you are getting required IOPS from storage or not.
2.) Universal forwarder sending more logs compare to indexers capacity. -> In this case if you are running Indexer cluster you need to add more indexer in your cluster.

Before increasing any queue size I'll recommnd to contact splunk support.

Thanks,
Harshil

[splunk24]

i am seeing the above issue for exec queue .. please guide what to do for that and exec queue is for running some scripts or what ?