How to configure DMC for heavy forwarder monitoring?

[koshyk]

hi,
we have few heavy forwarders which are used for intermediate forwarding to Indexers. In DMC (Distributed management console), we have got these forwarders correctly in the "Forwarders" Section and identified as "Heavy Forwarder" alongside 1000's of other external universal forwarders

But we are really looking to include it to be part of the core resource, so we can see "Resource Usage" section. Currently the resource Usage has "Cluster Master", "Indexer", "KV Store", SH, Deployment server but NO Heavy Forwarder.
Is there a way to include this into DMC to check resource utilisation etc on these heavy fwd's?

[codebuilder]

Forwarder monitoring already exists, and is available in two places actually. Both on the DMC and the Master nodes.
Monitoring from the Master requires indexer discovery to be enabled.

From either node go to Monitoring Console > Settings > Forwarder Monitoring Setup > rebuild forwarder assets.

Once this is done you'll be able to see all internal and external forwarders, as well as the type (light, universal, heavy), status, last communication date/time, data transfer rate, etc.

[codebuilder]

There are also "canned" forwarder alerts already available on the monitoring console (forwarder down, etc.)

[pellegrini]

Add the Heavy Forwarder as a search peer on the Monitoring Console host and assign server role Indexer to the Heavy Forwarder.
https://answers.splunk.com/answers/211976/distributed-managment-console-what-server-role-sho.html
There is no Heavy Forwarder role in DMC.

[koshyk]

Added as an idea: https://ideas.splunk.com/ideas/EID-I-73
Please upvote in there to speed up the feature

[inventsekar]

7.0.3 and still the same issue.
Anyone found out any workaround, please?!?!

[koshyk]

The workaround I'm doing is to install Splunk_TA_nix on the heavy-forwarders and extract the basic info. I've created similar dashboards (especially performance related metrics dashboards from them)

[koshyk]

Can anyone with Splunk enhancement request ability request this please? My permissions won't allow to raise Enhancement request nor ticket. thanks in advance.

[hettervik]

Plus 1, bump. As far as I can see this (simple) feature is still not included in the DMC.

[Kieffer87]

Nope, we just upgraded to 7.0.2 and still no HF role.

[Kieffer87]

Would also like to have the heavy forwarder role added to the DMC

[mathiask]

plus 1
Please add a Heavy Forwarder Role to DMC
I am using heavy forwarders for parsing events and load balancing them over the cluster
Advantages

• Split Parsing / Indexing load from Search load
• UFwd need no updates when the Cluster changes

But I lost visibility

[maciep]

Is there a reason you don't add the hf as a search peer on your dmc box? And then configure it as a Search Head in the setup (it technically is a search head after all). Just a thought. I feel like that might be easier than installing SoS on it.

Just looking how things work, the drop down on the resource usage instance page is based on server groups. So it would need to be part of a server group to show up there. If I had to guess, the reason is because DMC is using rest calls to pull the resource usage data for that page. And that would fail on most forwarders (universal forwarders for example), so they're not including the forwarders in that role selection.

[ryandg]

Doing this adds it as an indexer by default and has to be added to a specific tier-- there is no DMC role for HeavyForwarders so having it mixed in with your groups (especially large deployments) it can throw off/skew your metrics. I guess you could call them all deployment servers or something else if you don't want to monitor those servers or don't mind it being grouped together but honestly that's messier than just installing SOS in my opinion. I would rather just see an explicit role that can be assigned to Heavy Forwarders as their metrics on their own is important in monitoring their ingestion ability with CPU/Mem.

[koshyk]

that's another good option it seems. Theoretically it seems feasible, but need to try out. (want to keep away it from being a SH cluster though)

[ryandg]

I Me Too'd your post because I would much rather be able to do this on the DMC -- However I do have a work around. What we did is installed S.o.S on the HFs even though Splunk says you should use the DMC after 6.3 we needed to monitor CPU/Mem usage on our HFs that were consuming large amounts of Opsec traffic. Once SOS is installed we just monitored the HFs from the SearchHeads using the SoS. We didn't install the SOS TA anywhere other than the SHCs and the HFs so it doesn't need to run on all of your instances for it to work. It's not as nice as DMC granted but it will get the job done.

[koshyk]

Thanks mate for the workaround. I guess, we may need to raise it as an enhancement request to Splunk?

[tomasmoser]

HF is not A SH. It does index-time parsing except for indexing (should not). So it’s basically an indexer. Hence, give it IDX role in DMC/MC.

[ryandg]

Yeah that would be my best guess to get this added.