Deployment Architecture

Light Forwarder or switch to Universal Forwarder?

New Member

Currently we run syslog and linux monitor inputs into a linux indexer, but we monitor a few Windows boxes via UNC with a Forwarder (the Vendors don't want us running a Forwarder on the box along with their application). We just increased our licensed index amount and plan to add more Windows servers (maybe around 30), monitoring log files via UNC and I'd like to better understand what's going on and develop a plan to do it right.

I'm running the full Windows version of Splunk in Forwarder only mode (no indexing and/or searching). Is that the Light Forwarder?

Would I be better served to switch to the new Universal Forwarder at this time? Or should we just stick with the Light Forwarder? The Forwarder is running on a dedicated Quad Core box with 4 Megs of RAM. If I switched to the Universal Forwarder I'd put it on the same box.

Is there anything the Web interface provides which I couldn't do with CLI and *.conf files?

brew

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

To clarify

I'm running the full Windows version of Splunk in Forwarder only mode (no indexing and/or searching). Is that the Light Forwarder?

You are a LightWeightForwarder only if you installed a full splunk and enabled the app "SplunkLightForwarder" (using the UI switch or the app.conf)
Otherwise you are a "heavy forwarder" it means that you parse the events and either forward or index locally.

0 Karma

Splunk Employee
Splunk Employee

Heavy forwarder degrades the performance in terms of latency, i.e events getting indexed at indexer. Indexing time is huge when HF is used for forwarding the data. If the incoming data is at high speed, then you will be degrade of performance w.r.t 1). High index time 2). since HWF throughput is 4 times of input data, Indexer Queue will get filled faster unless your system has more than 4 core CPU for indexing. If Indexer Queue get filled, whole indexing functionality become slow.

Note: The recommended Indexer configuration is 12 Core CPU, 12 GB RAM and > 1200 IOPS.

Heavy Forwarder is recommended in the following scenarios
1. You want route the data to different indexer based on some patterns
2. Manipulating events with regex
3. You want to filter out lots of unwanted logs from input stream.

Since for your case, python is not required, Universal Forwarder suites best.

If you or any reader find this answer help in understanding your problem, please vote.

Regards
Jayanna Hallur
Wipro Technologies, Mountain View, California.

0 Karma

New Member

Hey, thanks dwaddle.

I'm going to leave it as the Heavy Forwarder - as time goes on we'll be adding more and more UNC inputs from Windows to it. I have the Forwarder on a dedicated server, I should use the computing power there rather than loading down my indexer!

And I'll see if I can fit in some Universal Forwarders on less critical Windows boxes as time goes on.

Thanks for the good info.

brew

0 Karma

New Member

Hey, thanks dwaddle.

I'm going to leave it as the Heavy Forwarder - as time goes on we'll be adding more and more UNC inputs from Windows to it. I have the Forwarder on a dedicated server, I should use the computing power there rather than loading down my indexer!

And I'll see if I can fit in some Universal Forwarders on less critical Windows boxes as time goes on.

Thanks for the good info.

brew

0 Karma

SplunkTrust
SplunkTrust

Currently, you could be running either the Light or Heavy forwarder, depending on which apps you have enabled. The Light forwarder uses the SplunkLightForwarder app, and the Heavy forwarder uses the SplunkForwarder app.

The Universal Forwarder is very similar to the Light Forwarder in terms of runtime footprint. (That is, CPU/memory/disk usage) Because it lacks the splunkweb web interface component it uses a few less resources. Its installation is also smaller on disk.

The fundamental difference between the Universal/Light forwarders and the Heavy forwarder is how they handle event parsing. The UF/LF do not do ANY event parsing, leaving this job for the indexers. The Heavy forwarder does basic event parsing locally before passing on "cooked" data to the indexer.

Given your description of your deployment, I'm not sure you really GAIN anything by going to UF - but I'm not sure you really LOSE anything either.

If you are comfortable in editing .conf files directly, the web interface isn't of much use to you -- because everything possible via the web interface is possib via a .conf file. In fact, the web interface is editing those .conf files on your behalf.

Given that your current windows forwarder is effectively a "dedicated forwarder machine" you may be happier with the full Heavy forwarder - as that offloads some work from your indexer. If you are going to actively deploy additional forwarder installations on your Windows servers I would highly recommend using the UF on those -- where you can of course. A forwarder on each node is more robust than pulling in logs via UNC path and enables capability that UNC collection does not like scripted inputs.