Getting Data In

Enqueuing a very large file on HF

bhsakarchourasi
Path Finder

Hi All,

My setup is firewall are sending logs to Syslog server and heavy forwarder installed on syslog server itself to read the files.

Since 2 days we are getting warn message "Enqueuing a very large file" and HF stopped sending logs to splunk cloud indexers (each file size is 2GB to 3.50GB in an hour).

Till now we tried increasing queue size that is set to unlimited in server.conf.
[queue=parsingQueue]
maxSize = 0

And also set thruput in limits.conf to unlimited.
[thruput]
maxKBps = 0

Please help to resolve this issue.

Thanks.
Bhaskar

0 Karma
1 Solution

ivanreis
Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

View solution in original post

0 Karma

torowa
Path Finder

I have a very similar scenario with large firewall logs going to Syslog server.
No issues with CPU but ingestion queued up and stalling.

The extra pipeline did the trick!  Thank you @ivanreis 

0 Karma

ivanreis
Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

0 Karma

bhsakarchourasi
Path Finder

Thanks a lot for you reply, first option worked and I revert all other changes that I made.

0 Karma
Get Updates on the Splunk Community!

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...