Getting Data In

Why is CPU on the universal forwarder not used most of the time and we see error "Could not send data to output queue (parsing queue), retrying...."?

earakam
Path Finder

Hi,

I was monitoring Universal Forwarder's CPU usage with the environment below, and I put 13GB sized file on Universal Forwarder server to send to the indexer and monitored it with limits.conf set maxKBps = 1024.

I noticed that most of the time, CPU isn't used, however, every 30 seconds it jumps up to about 10%.
I checked splunkd.log on the UF to see what's going on, and saw this error

Could not send data to output queue (parsing queue), retrying....

and at the same time, the the summary count on Indexer goes up.
It seems like the Universal Forwarder is sending data to Indexer only every 30 seconds?
Is there any interval for the UF to send data, or is the Indexer controlling UF from sending the data?

Indexer:
OS - Windows Server 2012 r2
Splunk - 6.3.2

Universal Forwarder:
OS - Centos64
Splunk - 6.3.2

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee
0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

earakam
Path Finder

Thanks for the answer!!

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Sounds like you are hitting your maxkbps limit. This would be logged.

The other possibility is to look at the IDX and make sure you are not blocking there.

index=_internal host=idxname source=*metrics.log blocked

But most likley you are hitting maxKBps. Is there a reason why you can't set it to 0 for unlimited?

0 Karma

earakam
Path Finder

Hi jwelch thanks for quick response!

i checked on the IDX and it i have found blocked=true every 10 minutes or so.
How do I stop IDX from blocking it?

The reason why i am doing this is just to see the CPU usage difference between when maxKBps = 0 and 1024.

Thakn you!

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...