Monitoring Splunk

How to configure DMC for heavy forwarder monitoring?

koshyk
Super Champion

hi,
we have few heavy forwarders which are used for intermediate forwarding to Indexers. In DMC (Distributed management console), we have got these forwarders correctly in the "Forwarders" Section and identified as "Heavy Forwarder" alongside 1000's of other external universal forwarders

But we are really looking to include it to be part of the core resource, so we can see "Resource Usage" section. Currently the resource Usage has "Cluster Master", "Indexer", "KV Store", SH, Deployment server but NO Heavy Forwarder.
Is there a way to include this into DMC to check resource utilisation etc on these heavy fwd's?

codebuilder
Influencer

Forwarder monitoring already exists, and is available in two places actually. Both on the DMC and the Master nodes.
Monitoring from the Master requires indexer discovery to be enabled.

From either node go to Monitoring Console > Settings > Forwarder Monitoring Setup > rebuild forwarder assets.

Once this is done you'll be able to see all internal and external forwarders, as well as the type (light, universal, heavy), status, last communication date/time, data transfer rate, etc.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
Influencer

There are also "canned" forwarder alerts already available on the monitoring console (forwarder down, etc.)

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

pellegrini
Path Finder

Add the Heavy Forwarder as a search peer on the Monitoring Console host and assign server role Indexer to the Heavy Forwarder.
https://answers.splunk.com/answers/211976/distributed-managment-console-what-server-role-sho.html
There is no Heavy Forwarder role in DMC.

koshyk
Super Champion

Added as an idea: https://ideas.splunk.com/ideas/EID-I-73
Please upvote in there to speed up the feature

inventsekar
Super Champion

7.0.3 and still the same issue.
Anyone found out any workaround, please?!?!

koshyk
Super Champion

The workaround I'm doing is to install Splunk_TA_nix on the heavy-forwarders and extract the basic info. I've created similar dashboards (especially performance related metrics dashboards from them)

0 Karma

koshyk
Super Champion

Can anyone with Splunk enhancement request ability request this please? My permissions won't allow to raise Enhancement request nor ticket. thanks in advance.

hettervi
Builder

Plus 1, bump. As far as I can see this (simple) feature is still not included in the DMC.

Kieffer87
Communicator

Nope, we just upgraded to 7.0.2 and still no HF role.

0 Karma

Kieffer87
Communicator

Would also like to have the heavy forwarder role added to the DMC

mathiask
Communicator

plus 1
Please add a Heavy Forwarder Role to DMC
I am using heavy forwarders for parsing events and load balancing them over the cluster
Advantages

  • Split Parsing / Indexing load from Search load
  • UFwd need no updates when the Cluster changes

But I lost visibility

maciep
Champion

Is there a reason you don't add the hf as a search peer on your dmc box? And then configure it as a Search Head in the setup (it technically is a search head after all). Just a thought. I feel like that might be easier than installing SoS on it.

Just looking how things work, the drop down on the resource usage instance page is based on server groups. So it would need to be part of a server group to show up there. If I had to guess, the reason is because DMC is using rest calls to pull the resource usage data for that page. And that would fail on most forwarders (universal forwarders for example), so they're not including the forwarders in that role selection.

ryandg
Communicator

Doing this adds it as an indexer by default and has to be added to a specific tier-- there is no DMC role for HeavyForwarders so having it mixed in with your groups (especially large deployments) it can throw off/skew your metrics. I guess you could call them all deployment servers or something else if you don't want to monitor those servers or don't mind it being grouped together but honestly that's messier than just installing SOS in my opinion. I would rather just see an explicit role that can be assigned to Heavy Forwarders as their metrics on their own is important in monitoring their ingestion ability with CPU/Mem.

koshyk
Super Champion

that's another good option it seems. Theoretically it seems feasible, but need to try out. (want to keep away it from being a SH cluster though)

0 Karma

ryandg
Communicator

I Me Too'd your post because I would much rather be able to do this on the DMC -- However I do have a work around. What we did is installed S.o.S on the HFs even though Splunk says you should use the DMC after 6.3 we needed to monitor CPU/Mem usage on our HFs that were consuming large amounts of Opsec traffic. Once SOS is installed we just monitored the HFs from the SearchHeads using the SoS. We didn't install the SOS TA anywhere other than the SHCs and the HFs so it doesn't need to run on all of your instances for it to work. It's not as nice as DMC granted but it will get the job done.

koshyk
Super Champion

Thanks mate for the workaround. I guess, we may need to raise it as an enhancement request to Splunk?

0 Karma

tomasmoser
Contributor

HF is not A SH. It does index-time parsing except for indexing (should not). So it’s basically an indexer. Hence, give it IDX role in DMC/MC.

0 Karma

ryandg
Communicator

Yeah that would be my best guess to get this added.

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!