I have 4 universal forwarders set up in a DMZ that receive events from other universal forwarders in the field and relay them to our indexers (so as not to expose our indexers directly to the internet).
I noticed that each quad-cpu forwarder in the DMZ has utilization on only a single CPU and only 1 splunkd.exe process.
Is this a limitation of the application, or is it configurable? I didn't want to install a full Splunk instance because I wanted to limit our DMZ footprint and don't want the relay hosts to count against our indexing license. If I were to replace a UF with a full Splunk installation that has no indexes set up and the same inputs.conf and outputs.conf, would that allow us to make use of multiple CPUs?
Considering the dual-cores are recommended by splunk for UF and "light" forwarder, I would think that the app is can fully utilize multiple CPUs/Cores. On linux top and ps show multiple threads if you tell the tool to show threads. If you are not seeing the the throughput/utilization you expect, perhaps there is some other bottleneck. The first place I would look on a UF is in limits.conf, especially at "thruput" since the default on a UF is 256kbps if I recall correctly.
Our throughput seems fine, 1 CPU on each host is at about 25% and events seems current. I was just surprised to see only 1 process and only 1 CPU being utilized on each host with 6000 UF's sending in events. We expect to start receiving from an additional 8000 machines in the field over the next week and I am trying to gauge resources. If the additional cores are not being used I will take them away and save 12 cores (these are VMs).
Right, you might want to reassess when you start ramping up your forwarding. 12 cores for a UF is pretty huge. At least with VMs you can safely oversubscribe if they are not actually using them, but there is no sense in total overkill.
Also note, that if you use a full splunk instance as your forwarders here, you will not incur license cost unless these systems were configured to index the same data that you have them forward. You can also disable splunk web on heavy forwarders through the settings page or in web.conf with the following if you decide that you need the capabilities of a heavy forwarder:
startwebserver = 0