Monitoring Splunk

How to configure DMC for heavy forwarder monitoring?

Super Champion

hi,
we have few heavy forwarders which are used for intermediate forwarding to Indexers. In DMC (Distributed management console), we have got these forwarders correctly in the "Forwarders" Section and identified as "Heavy Forwarder" alongside 1000's of other external universal forwarders

But we are really looking to include it to be part of the core resource, so we can see "Resource Usage" section. Currently the resource Usage has "Cluster Master", "Indexer", "KV Store", SH, Deployment server but NO Heavy Forwarder.
Is there a way to include this into DMC to check resource utilisation etc on these heavy fwd's?

Motivator

Forwarder monitoring already exists, and is available in two places actually. Both on the DMC and the Master nodes.
Monitoring from the Master requires indexer discovery to be enabled.

From either node go to Monitoring Console > Settings > Forwarder Monitoring Setup > rebuild forwarder assets.

Once this is done you'll be able to see all internal and external forwarders, as well as the type (light, universal, heavy), status, last communication date/time, data transfer rate, etc.

0 Karma

Motivator

There are also "canned" forwarder alerts already available on the monitoring console (forwarder down, etc.)

0 Karma

Explorer

Add the Heavy Forwarder as a search peer on the Monitoring Console host and assign server role Indexer to the Heavy Forwarder.
https://answers.splunk.com/answers/211976/distributed-managment-console-what-server-role-sho.html
There is no Heavy Forwarder role in DMC.

0 Karma

Super Champion

Added as an idea: https://ideas.splunk.com/ideas/EID-I-73
Please upvote in there to speed up the feature

Champion

7.0.3 and still the same issue.
Anyone found out any workaround, please?!?!

Super Champion

The workaround I'm doing is to install SplunkTAnix on the heavy-forwarders and extract the basic info. I've created similar dashboards (especially performance related metrics dashboards from them)

0 Karma

Super Champion

Can anyone with Splunk enhancement request ability request this please? My permissions won't allow to raise Enhancement request nor ticket. thanks in advance.

Builder

Plus 1, bump. As far as I can see this (simple) feature is still not included in the DMC.

Communicator

Nope, we just upgraded to 7.0.2 and still no HF role.

0 Karma

Communicator

Would also like to have the heavy forwarder role added to the DMC

Communicator

plus 1
Please add a Heavy Forwarder Role to DMC
I am using heavy forwarders for parsing events and load balancing them over the cluster
Advantages

  • Split Parsing / Indexing load from Search load
  • UFwd need no updates when the Cluster changes

But I lost visibility

Champion

Is there a reason you don't add the hf as a search peer on your dmc box? And then configure it as a Search Head in the setup (it technically is a search head after all). Just a thought. I feel like that might be easier than installing SoS on it.

Just looking how things work, the drop down on the resource usage instance page is based on server groups. So it would need to be part of a server group to show up there. If I had to guess, the reason is because DMC is using rest calls to pull the resource usage data for that page. And that would fail on most forwarders (universal forwarders for example), so they're not including the forwarders in that role selection.

Communicator

Doing this adds it as an indexer by default and has to be added to a specific tier-- there is no DMC role for HeavyForwarders so having it mixed in with your groups (especially large deployments) it can throw off/skew your metrics. I guess you could call them all deployment servers or something else if you don't want to monitor those servers or don't mind it being grouped together but honestly that's messier than just installing SOS in my opinion. I would rather just see an explicit role that can be assigned to Heavy Forwarders as their metrics on their own is important in monitoring their ingestion ability with CPU/Mem.

Super Champion

that's another good option it seems. Theoretically it seems feasible, but need to try out. (want to keep away it from being a SH cluster though)

0 Karma

Communicator

I Me Too'd your post because I would much rather be able to do this on the DMC -- However I do have a work around. What we did is installed S.o.S on the HFs even though Splunk says you should use the DMC after 6.3 we needed to monitor CPU/Mem usage on our HFs that were consuming large amounts of Opsec traffic. Once SOS is installed we just monitored the HFs from the SearchHeads using the SoS. We didn't install the SOS TA anywhere other than the SHCs and the HFs so it doesn't need to run on all of your instances for it to work. It's not as nice as DMC granted but it will get the job done.

Super Champion

Thanks mate for the workaround. I guess, we may need to raise it as an enhancement request to Splunk?

0 Karma

Contributor

HF is not A SH. It does index-time parsing except for indexing (should not). So it’s basically an indexer. Hence, give it IDX role in DMC/MC.

0 Karma

Communicator

Yeah that would be my best guess to get this added.

0 Karma