Getting Data In

Why is CPU on the universal forwarder not used most of the time and we see error "Could not send data to output queue (parsing queue), retrying...."?

earakam
Path Finder

Hi,

I was monitoring Universal Forwarder's CPU usage with the environment below, and I put 13GB sized file on Universal Forwarder server to send to the indexer and monitored it with limits.conf set maxKBps = 1024.

I noticed that most of the time, CPU isn't used, however, every 30 seconds it jumps up to about 10%.
I checked splunkd.log on the UF to see what's going on, and saw this error

Could not send data to output queue (parsing queue), retrying....

and at the same time, the the summary count on Indexer goes up.
It seems like the Universal Forwarder is sending data to Indexer only every 30 seconds?
Is there any interval for the UF to send data, or is the Indexer controlling UF from sending the data?

Indexer:
OS - Windows Server 2012 r2
Splunk - 6.3.2

Universal Forwarder:
OS - Centos64
Splunk - 6.3.2

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee
0 Karma

yannK
Splunk Employee
Splunk Employee
0 Karma

earakam
Path Finder

Thanks for the answer!!

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Sounds like you are hitting your maxkbps limit. This would be logged.

The other possibility is to look at the IDX and make sure you are not blocking there.

index=_internal host=idxname source=*metrics.log blocked

But most likley you are hitting maxKBps. Is there a reason why you can't set it to 0 for unlimited?

0 Karma

earakam
Path Finder

Hi jwelch thanks for quick response!

i checked on the IDX and it i have found blocked=true every 10 minutes or so.
How do I stop IDX from blocking it?

The reason why i am doing this is just to see the CPU usage difference between when maxKBps = 0 and 1024.

Thakn you!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...