Getting Data In

Enqueuing a very large file on HF

New Member

Hi All,

My setup is firewall are sending logs to Syslog server and heavy forwarder installed on syslog server itself to read the files.

Since 2 days we are getting warn message "Enqueuing a very large file" and HF stopped sending logs to splunk cloud indexers (each file size is 2GB to 3.50GB in an hour).

Till now we tried increasing queue size that is set to unlimited in server.conf.
[queue=parsingQueue]
maxSize = 0

And also set thruput in limits.conf to unlimited.
[thruput]
maxKBps = 0

Please help to resolve this issue.

Thanks.
Bhaskar

0 Karma
1 Solution

Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

View solution in original post

0 Karma

Builder

Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...

Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.

Please avoid to setup the configuration to unlimited, this is not the path to move on.

Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.

Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.

View solution in original post

0 Karma

New Member

Thanks a lot for you reply, first option worked and I revert all other changes that I made.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!