Getting Data In

Enqueuing a very large file on HF

bhsakarchourasi
Path Finder

Hi All,

My setup is firewall are sending logs to Syslog server and heavy forwarder installed on syslog server itself to read the files.

Since 2 days we are getting warn message "Enqueuing a very large file" and HF stopped sending logs to splunk cloud indexers (each file size is 2GB to 3.50GB in an hour).

Till now we tried increasing queue size that is set to unlimited in server.conf.
[queue=parsingQueue]
maxSize = 0

And also set thruput in limits.conf to unlimited.
[thruput]
maxKBps = 0

Please help to resolve this issue.

Thanks.
Bhaskar

0 Karma
1 Solution

ivanreis
Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

View solution in original post

0 Karma

torowa
Path Finder

I have a very similar scenario with large firewall logs going to Syslog server.
No issues with CPU but ingestion queued up and stalling.

The extra pipeline did the trick!  Thank you @ivanreis 

0 Karma

ivanreis
Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

0 Karma

bhsakarchourasi
Path Finder

Thanks a lot for you reply, first option worked and I revert all other changes that I made.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...