Why is CPU on the universal forwarder not used most of the time and we see error "Could not send data to output queue (parsing queue), retrying...."?

[earakam]

Hi,

I was monitoring Universal Forwarder's CPU usage with the environment below, and I put 13GB sized file on Universal Forwarder server to send to the indexer and monitored it with limits.conf set maxKBps = 1024.

I noticed that most of the time, CPU isn't used, however, every 30 seconds it jumps up to about 10%.
I checked splunkd.log on the UF to see what's going on, and saw this error

Could not send data to output queue (parsing queue), retrying....

and at the same time, the the summary count on Indexer goes up.
It seems like the Universal Forwarder is sending data to Indexer only every 30 seconds?
Is there any interval for the UF to send data, or is the Indexer controlling UF from sending the data?

Indexer:
OS - Windows Server 2012 r2
Splunk - 6.3.2

Universal Forwarder:
OS - Centos64
Splunk - 6.3.2

[yannK · Accepted solution]

check your thruput in limits.conf on the forwarder

see http://docs.splunk.com/Documentation/Splunk/6.3.3/Troubleshooting/Troubleshootingeventsindexingdelay...

[earakam]

Thanks for the answer!!

[jwelch_splunk]

Sounds like you are hitting your maxkbps limit. This would be logged.

The other possibility is to look at the IDX and make sure you are not blocking there.

index=_internal host=idxname source=*metrics.log blocked

But most likley you are hitting maxKBps. Is there a reason why you can't set it to 0 for unlimited?

[earakam]

Hi jwelch thanks for quick response!

i checked on the IDX and it i have found blocked=true every 10 minutes or so.
How do I stop IDX from blocking it?

The reason why i am doing this is just to see the CPU usage difference between when maxKBps = 0 and 1024.

Thakn you!