we have few heavy forwarders which are used for intermediate forwarding to Indexers. In DMC (Distributed management console), we have got these forwarders correctly in the "Forwarders" Section and identified as "Heavy Forwarder" alongside 1000's of other external universal forwarders
But we are really looking to include it to be part of the core resource, so we can see "Resource Usage" section. Currently the resource Usage has "Cluster Master", "Indexer", "KV Store", SH, Deployment server but NO Heavy Forwarder.
Is there a way to include this into DMC to check resource utilisation etc on these heavy fwd's?
Forwarder monitoring already exists, and is available in two places actually. Both on the DMC and the Master nodes.
Monitoring from the Master requires indexer discovery to be enabled.
From either node go to Monitoring Console > Settings > Forwarder Monitoring Setup > rebuild forwarder assets.
Once this is done you'll be able to see all internal and external forwarders, as well as the type (light, universal, heavy), status, last communication date/time, data transfer rate, etc.
Add the Heavy Forwarder as a search peer on the Monitoring Console host and assign server role Indexer to the Heavy Forwarder.
There is no Heavy Forwarder role in DMC.
The workaround I'm doing is to install Splunk_TA_nix on the heavy-forwarders and extract the basic info. I've created similar dashboards (especially performance related metrics dashboards from them)
Please add a Heavy Forwarder Role to DMC
I am using heavy forwarders for parsing events and load balancing them over the cluster
But I lost visibility
Is there a reason you don't add the hf as a search peer on your dmc box? And then configure it as a Search Head in the setup (it technically is a search head after all). Just a thought. I feel like that might be easier than installing SoS on it.
Just looking how things work, the drop down on the resource usage instance page is based on server groups. So it would need to be part of a server group to show up there. If I had to guess, the reason is because DMC is using rest calls to pull the resource usage data for that page. And that would fail on most forwarders (universal forwarders for example), so they're not including the forwarders in that role selection.
Doing this adds it as an indexer by default and has to be added to a specific tier-- there is no DMC role for HeavyForwarders so having it mixed in with your groups (especially large deployments) it can throw off/skew your metrics. I guess you could call them all deployment servers or something else if you don't want to monitor those servers or don't mind it being grouped together but honestly that's messier than just installing SOS in my opinion. I would rather just see an explicit role that can be assigned to Heavy Forwarders as their metrics on their own is important in monitoring their ingestion ability with CPU/Mem.
I Me Too'd your post because I would much rather be able to do this on the DMC -- However I do have a work around. What we did is installed S.o.S on the HFs even though Splunk says you should use the DMC after 6.3 we needed to monitor CPU/Mem usage on our HFs that were consuming large amounts of Opsec traffic. Once SOS is installed we just monitored the HFs from the SearchHeads using the SoS. We didn't install the SOS TA anywhere other than the SHCs and the HFs so it doesn't need to run on all of your instances for it to work. It's not as nice as DMC granted but it will get the job done.