Getting Data In

Why is CPU on the universal forwarder not used most of the time and we see error "Could not send data to output queue (parsing queue), retrying...."?

Path Finder

Hi,

I was monitoring Universal Forwarder's CPU usage with the environment below, and I put 13GB sized file on Universal Forwarder server to send to the indexer and monitored it with limits.conf set maxKBps = 1024.

I noticed that most of the time, CPU isn't used, however, every 30 seconds it jumps up to about 10%.
I checked splunkd.log on the UF to see what's going on, and saw this error

Could not send data to output queue (parsing queue), retrying....

and at the same time, the the summary count on Indexer goes up.
It seems like the Universal Forwarder is sending data to Indexer only every 30 seconds?
Is there any interval for the UF to send data, or is the Indexer controlling UF from sending the data?

Indexer:
OS - Windows Server 2012 r2
Splunk - 6.3.2

Universal Forwarder:
OS - Centos64
Splunk - 6.3.2

0 Karma
1 Solution

Splunk Employee
Splunk Employee
0 Karma

Splunk Employee
Splunk Employee
0 Karma

Path Finder

Thanks for the answer!!

0 Karma

Splunk Employee
Splunk Employee

Sounds like you are hitting your maxkbps limit. This would be logged.

The other possibility is to look at the IDX and make sure you are not blocking there.

index=_internal host=idxname source=*metrics.log blocked

But most likley you are hitting maxKBps. Is there a reason why you can't set it to 0 for unlimited?

0 Karma

Path Finder

Hi jwelch thanks for quick response!

i checked on the IDX and it i have found blocked=true every 10 minutes or so.
How do I stop IDX from blocking it?

The reason why i am doing this is just to see the CPU usage difference between when maxKBps = 0 and 1024.

Thakn you!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!