Hi All,
My setup is firewall are sending logs to Syslog server and heavy forwarder installed on syslog server itself to read the files.
Since 2 days we are getting warn message "Enqueuing a very large file" and HF stopped sending logs to splunk cloud indexers (each file size is 2GB to 3.50GB in an hour).
Till now we tried increasing queue size that is set to unlimited in server.conf.
[queue=parsingQueue]
maxSize = 0
And also set thruput in limits.conf to unlimited.
[thruput]
maxKBps = 0
Please help to resolve this issue.
Thanks.
Bhaskar
Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...
Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.
Please avoid to setup the configuration to unlimited, this is not the path to move on.
Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.
Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.
I have a very similar scenario with large firewall logs going to Syslog server.
No issues with CPU but ingestion queued up and stalling.
The extra pipeline did the trick! Thank you @ivanreis
Check in your forwarder if the CPU is not overloaded, you can create a 2nd pipeline on the HF. It will assist to HF to parse more data
Here is the process to create a 2nd pipeline.
https://docs.splunk.com/Documentation/Forwarder/7.3.2/Forwarder/Configureaforwardertohandlemultiplep...
Verify the possibility to decrease the size of the file, maybe you can use the regex to filter out unnecessary to the null queue.
Please avoid to setup the configuration to unlimited, this is not the path to move on.
Maybe you should think about to add more heavy forwarders to behind the load balancer to increase the ingestion capacity and add 2 or 3 HF to the tier.
Other potential solution, is to forward the data direct to the indexers if the data you are indexing does not require the HF. Doing this you can increase the indexing capability to spread the data to other indexers instead using only 1 server.
Thanks a lot for you reply, first option worked and I revert all other changes that I made.