https://community.splunk.com/t5/Splunk-Enterprise/Resource-issues-with-indexing-and-queues/m-p/540795#M5011 <P>Hi all, so I am facing this issue with what seems to be delayed/not receiving logs from the UFs.</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="d_lim_1-1613984409525.png" style="width: 400px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/13001i349B8190842FB37D/image-size/medium?v=v2&amp;px=400" role="button" title="d_lim_1-1613984409525.png" alt="d_lim_1-1613984409525.png" /></span></P><P>This is the current queue, we have gone from 1, to 2, to now 3 for the indexer's parallelIngestionPipeline settings.</P><P>On the indexer, the index queue is always full and is affecting the downstream from the 2 HFs.&nbsp;</P><P>There are about 16 intermediate forwarders sending to HF001, and HF002 is mainly doing API calls to pull data.</P><P>The iops for the indexer is around 1600, cpu usage 50% and memory 31%.</P><P>Any recommendations on what we can do to improve this, eg. additional indexer? Thanks.</P> Mon, 22 Feb 2021 09:02:05 GMT d_lim 2021-02-22T09:02:05Z https://community.splunk.com/t5/Splunk-Enterprise/Resource-issues-with-indexing-and-queues/m-p/540795#M5011 <P>Hi all, so I am facing this issue with what seems to be delayed/not receiving logs from the UFs.</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="d_lim_1-1613984409525.png" style="width: 400px;"><img src="https://community.splunk.com/t5/image/serverpage/image-id/13001i349B8190842FB37D/image-size/medium?v=v2&amp;px=400" role="button" title="d_lim_1-1613984409525.png" alt="d_lim_1-1613984409525.png" /></span></P><P>This is the current queue, we have gone from 1, to 2, to now 3 for the indexer's parallelIngestionPipeline settings.</P><P>On the indexer, the index queue is always full and is affecting the downstream from the 2 HFs.&nbsp;</P><P>There are about 16 intermediate forwarders sending to HF001, and HF002 is mainly doing API calls to pull data.</P><P>The iops for the indexer is around 1600, cpu usage 50% and memory 31%.</P><P>Any recommendations on what we can do to improve this, eg. additional indexer? Thanks.</P> Mon, 22 Feb 2021 09:02:05 GMT https://community.splunk.com/t5/Splunk-Enterprise/Resource-issues-with-indexing-and-queues/m-p/540795#M5011 d_lim 2021-02-22T09:02:05Z https://community.splunk.com/t5/Splunk-Enterprise/Resource-issues-with-indexing-and-queues/m-p/540839#M5013 <P>I full indexing queue means the act of writing to disk is taking too long.&nbsp; Adding pipelines just makes that worse by creating more threads that try to write to disk.&nbsp; Something in the storage system is causing delays and correcting that problem should alleviate the queue problem.</P> Mon, 22 Feb 2021 14:02:06 GMT https://community.splunk.com/t5/Splunk-Enterprise/Resource-issues-with-indexing-and-queues/m-p/540839#M5013 richgalloway 2021-02-22T14:02:06Z