Getting Data In

Universal Forwarder: Why are the High-volume log files are forwarding at at a ridiculously low level?

chrismallow
Engager

I'm running a Bro sensor with some (obviously) very high-volume log files that I'm monitoring with the Universal Forwarder. Some of these files are adding events at 500-2000 events/s. The forwarder is forwarding, but at a ridiculously low level, around 5-30 events/s. This is obviously not correct.

I initially set this up via forwarder management in the UI. I have tried both monitorings of all '.log' files in the target directory (/opt/bro/spool/bro), and during troubleshooting, even tried to isolate individual files with their own inputs (e.g., /opt/bro/spool/bro/conn.log). I have also added items directly to the 'inputs.conf' file on the server.

I've researched several options and the only one I've found that could possibly be relevant is the 'crcSalt' option. I've tried that but with no success. Nothing is working to get the forwarder to send these at the actual volume in real-time speed, as I would expect. My inputs.conf currently looks like this:

[monitor:///opt/bro/spool/bro/conn.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt = <SOURCE>

[monitor:///opt/bro/spool/bro/dns.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt = <SOURCE>

[monitor:///opt/bro/spool/bro/http.log]
disabled = false
index = bro-log
sourcetype = bro
crcSalt = <SOURCE>

If I can't make this work, I will simply ditch the Universal Forwarder and go back to the rsyslog forwarding method I had previously that worked perfectly. Any thoughts or suggestions would be greatly appreciated.

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi chrismallow,
you could increase your throughput but you will consume more network bandwidht.
The way to do this is to modify in your Universal Forwarder the value maxKBps in limits.conf.
You can find more datails at https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Limitsconf .

I suggest to follow indication described at https://answers.splunk.com/answers/134801/how-do-you-know-when-to-increase-the-bandwidth-limits.html : in this answer he suggested to increase the bandwidht consuption but also to check it.

You can check it using the Universal Forwarder's logs that you can find on the machine in $SPLUNK_HOME/var/log/splunk/metrics.log or in Splunk using a simple search

index=_internal source="*metrics.log" group=thruput

Bye.
Giuseppe

View solution in original post

FrankVl
Ultra Champion

Universal Forwarders are by default limited to 256kBps, so make sure to adjust the limits.conf to either completely remove the limit (set it to 0) or set it to a suitable value as @cusello is already mentioning in his answer.

While doing so, make sure to keep an eye on the queues throughout your architecture, to see if there is any congestion happening that is causing poor throughput.

Note: that crcSalt setting has no effect if your monitor stanza is looking at specific filenames (as setting it to SOURCE simply adds the filepath/name to the crc function).

Another thing to keep in mind: with such high volume files, Splunk will likely have trouble getting the autoLB feature to work properly (assuming you are using that to distribute events over multiple indexers). So you might want to take a look some of the more advanced features that were introduced in Splunk 6.5, to enable Universal Fowarders to recognize event boundaries: http://docs.splunk.com/Documentation/Forwarder/7.0.2/Forwarder/Configureloadbalancing

nickhills
Ultra Champion

Its also possible you have contention on your queues.

If events were being sent 'faster' over syslog (and they still contained the same volume of data) its quite possible the the bottleneck is occurring elsewhere in your deployment, like the parsing queue.

Take a look with something like:
index=_internal sourcetype=splunkd group=queue blocked

If my comment helps, please give it a thumbs up!
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi chrismallow,
you could increase your throughput but you will consume more network bandwidht.
The way to do this is to modify in your Universal Forwarder the value maxKBps in limits.conf.
You can find more datails at https://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Limitsconf .

I suggest to follow indication described at https://answers.splunk.com/answers/134801/how-do-you-know-when-to-increase-the-bandwidth-limits.html : in this answer he suggested to increase the bandwidht consuption but also to check it.

You can check it using the Universal Forwarder's logs that you can find on the machine in $SPLUNK_HOME/var/log/splunk/metrics.log or in Splunk using a simple search

index=_internal source="*metrics.log" group=thruput

Bye.
Giuseppe

chrismallow
Engager

This (and FrankVI's below) was the correct answer! Setting 'MaxKBps = 0' resolved the problem.

I can't believe I didn't find this setting...I hunted through the install docs and about 20 different answers articles trying to figure out what was going on. Thank you all for your help!!

0 Karma

shaikhussain2
Explorer

Hi Chrismallow,

Same issue i am facing now, can you please help me out how you have resolved this issue!!

0 Karma

stboch
SplunkTrust
SplunkTrust

Additional recommendation, if your logs are really high volume you may want to increase the number of parallel pipelines.

server.conf
[general]
parallelIngestionPipelines=(more than 1)

Be careful to watch system resources, as it is basically x increase the system workload. Most times just increasing this to 2 is enough to increase the performance.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...