Getting Data In

Why is CPU on the universal forwarder not used most of the time and we see error "Could not send data to output queue (parsing queue), retrying...."?

earakam
Path Finder

Hi,

I was monitoring Universal Forwarder's CPU usage with the environment below, and I put 13GB sized file on Universal Forwarder server to send to the indexer and monitored it with limits.conf set maxKBps = 1024.

I noticed that most of the time, CPU isn't used, however, every 30 seconds it jumps up to about 10%.
I checked splunkd.log on the UF to see what's going on, and saw this error

Could not send data to output queue (parsing queue), retrying....

and at the same time, the the summary count on Indexer goes up.
It seems like the Universal Forwarder is sending data to Indexer only every 30 seconds?
Is there any interval for the UF to send data, or is the Indexer controlling UF from sending the data?

Indexer:
OS - Windows Server 2012 r2
Splunk - 6.3.2

Universal Forwarder:
OS - Centos64
Splunk - 6.3.2

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee
0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

earakam
Path Finder

Thanks for the answer!!

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Sounds like you are hitting your maxkbps limit. This would be logged.

The other possibility is to look at the IDX and make sure you are not blocking there.

index=_internal host=idxname source=*metrics.log blocked

But most likley you are hitting maxKBps. Is there a reason why you can't set it to 0 for unlimited?

0 Karma

earakam
Path Finder

Hi jwelch thanks for quick response!

i checked on the IDX and it i have found blocked=true every 10 minutes or so.
How do I stop IDX from blocking it?

The reason why i am doing this is just to see the CPU usage difference between when maxKBps = 0 and 1024.

Thakn you!

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...