Queue Constantly being full

[KJL]

Hello everyone! I am a new splunk user and I am noticing that my splunk HF is constantly having a high p90 queue fill perc. 

I ran the following search index=_internal host=<myhost> blocked=true and I am seeing max_size_kb of 500 - 10240 getting block. If I am not wrong, throughput for a HF is set at 256KBps.

I looked into the server that is running the HF, but it does seem that seem to be having any high CPU/IOPS usage, is there any way I can troubleshoot this? 

[isoutamo]

Here is excellent conf presentation how to find why queue is full. https://conf.splunk.com/files/2019/slides/FN1570.pdf

[PickleRick]

* Default (Splunk Enterprise): 0 (unlimited)
* Default (Splunk Universal Forwarder): 256

So for a HF it's not a thruput cap issue (unless you reconfigured your limits). If your outputs are blocking look downstream. Either for network problems or clogged indexers. 

[KJL]

Thank you for your replies @livehybrid  @PrewinThomas 

index=_internal source="*license_usage.log" type=Usage h="<forwader name>"

| rename _time as Date

| eval Date=strftime(Date,"%b-%y")

| stats sum(b) as license by Date h

| eval licenseGB =round(license/1024/1024/1024,3)

| rename licenseGB as TB

How to calculate data ingestion from a specific He... - Splunk Community

using this search from the community, it seems that my heavy forwarder with throttling issue is forwarding about 16-28 GB daily as opposed to another heavy forwarder forwarding about >2GB daily. Currently in the limits.conf file, throughput rate is configured at 0 (hence no limit). Is there any way I can still configure the heavy forwarder to take on such a load of 16GB daily? 

Increasing the hardware on the heavy forwarder did not seem to do the trick so not sure if I can reconfigure the heavy forwarder limits. Additionally, is 16GB too high? not sure what is the benchmark for this as I am rather new. 

Answering the question on the queues that were getting choked up, "1 - Parsing Queue 2 - Aggregation Queue 3 - Typing Queue 4 - Indexing Queue 5 - TcpOut Queue", all 5 queues are constantly at near 100%. 
query used: 

index=_internal source=*metrics.log sourcetype=splunkd group=queue (name=parsingqueue OR name=aggqueue OR name=typingqueue OR name=indexqueue OR name=tcpout* OR name=tcpin_queue) host IN (<your host>)

| replace tcpout* with tcpoutqueue in name

| eval name=case(name=="tcpin_queue","0 - TcpIn Queue",name=="aggqueue","2 - Aggregation Queue",name=="indexqueue","4 - Indexing Queue",name=="parsingqueue","1 - Parsing Queue",name=="typingqueue","3 - Typing Queue",name=="tcpoutqueue","5 - TcpOut Queue")

| eval max=if(isnotnull(max_size_kb),max_size_kb,max_size)

| eval curr=if(isnotnull(current_size_kb),current_size_kb,current_size)

| eval fill_perc=round((curr/max)*100,2)

| timechart span=30m p90(fill_perc) AS fill_perc by name

[livehybrid]

Hi @KJL 

What is the name of the queue that is being blocked?

Do you know the amount of data being sent to this instance? (is the load spread across other HFs?) 

Have you recently installed/updated any apps or applied new config, or increased ingestion?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PrewinThomas]

@KJL 

You mentioned seeing blocked=true in _internal logs with max_size_kb ranging from 500 to 10240. That’s a sign that Splunk is throttling because the queues are full.

If your HF is set at 256KBps, which can be a bottleneck if you're forwarding a lot of data.

To start with, try increase that to 2048 or 0(no cap) depending on your system’s/network capacity.

Also verify your connectivity towards receiving end(intermediate HF/Indexer). If there is n/w latency or slow performance at receiving end, then queues will back up.

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!