Splunk Enterprise Parallel Data Pipelines on the HF are full after upgrading to verstion 9.0.0

[muradgh]

Hi All,

Recently I have upgraded Splunk to the latest version (9.0.0) on the DS & HF & AIO machines I have, everything was working just fine before upgrading anything, after upgrading the whole set of machines things went wrong, the main problem is in the HF, in the "Health Status of Splunkd" the "File Monitor Input" sign is Red for almost all of them as shown in the screenshot below: 

Besides the following messages:

I have noticed that in the "Monitoring Console" -> "Indexing" -> "Performance" -> "Indexing Performance: Instance" the queue fill ratio is 100% on all the pipelines as shown in the screenshot below:

The server itself is not utilized, it has the following specs:

OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
CPU: 8 cores
RAM: 64 GB

Can anybody lead me to what is the cause of this problem?
Much thanks
Murad Ghazzawi.

[dokaas_2]

I'm seeing something similar with my v9 intermediate forwarders.  I have not upgraded my indexer cluster to v9 yet (8.2.5) and was wondering if the forwarders being at a higher version than the indexer was causing the issues. 

I'm seeing events from the problem relay servers such as:

   Invalid ACK received from indexer=aaa.bbb.ccc.ddd:9997. Got unexpected ACK with eventid=53825

Are you seeing anything such as that?

  index=_internal sourcetype=splunkd ack