Monitoring Splunk
Highlighted

Is the Monitoring Console (MC) tool really worth the effort of implementing?

Path Finder

Hello Splunk Admins,

I am seeking your opinion on the worth of implementing the MC tool as a way to help monitor my splunk infrastructure.

I have read the pages and pages and the even more confusing nested pages on what it takes to get it off the ground. Just reading WHERE to install it was confusing enough. My heads hurts after this and am a bit apprehensive. I don't need any more rabbit holes to go down into at the moment. ;-).

*Concerns *
I found the DMC tool (predecessor to MC) already present on scattered nodes in our environment.
1. How do I migrate from DMC tool to MC tool? Do I remove the existing DMC on nodes where found?
2. Does MC actually run on V6.4.1 splunk enterprise or do I need to upgrade to 6.5 and above?
3. Should I stay with DMC and avoid the hassle of switching over to MC?

*Background info *
Splunk Enterprise V6.4.1
All nodes are RHEL 6.8 and 7.3

1 SH cluster consisting of 3 SH members.
1 SH deployer that manages the SH cluster
3 Indexer clusters (one cluster per Data Center)
3 Master nodes that manage each indexer cluster. So 1 MN per Data center.
1 deployment server for App management

Cutting to the chase, can't I just:
Option #1:
1. spin up a new VM w/ plenty of resources(ram,cpus, etc)
2. Install splunk on it.
3. configure it as a SH member as per the doc
4. Install MC on it
5. Configure it and off we go.

Or Option #2:
1. Install Splunk add-on for *nix on each node, setup a few Alerts, create a Dashboard and off we go

Or Option #3:
1. Use scripts I wrote that monitor indexer space usage, system performance, other basics, etc.

Please advise me on my questions stated above. All help is appreciated.

Sorry for the long info, but wanted to give a clear description of my current environment and avoid creating a mess.
cheers,
Damon

Highlighted

Re: Is the Monitoring Console (MC) tool really worth the effort of implementing?

SplunkTrust
SplunkTrust

I believe there's nothing weirdly hard about just doing your option 1.

But, why not upgrade?

I haven't done SHC or deployer, but the rest are - well, my last upgrade from 6.5 to 6.6 was in a very similar but slightly smaller environment (except the SHC stuff) and was finished in about an hour. You could probably do your own in less than 2.

If you chose to go this route, read the upgrade docs to at least go to 6.6, if not 7. And the release notes. If you are even a TINY bit uncertain, spin up a quick test lab (it needs almost no data) to just try it out. And obviously I'm not responsible for any dead parakeets (or kicked puppies, or servers misbehaving... whatever).

Highlighted

Re: Is the Monitoring Console (MC) tool really worth the effort of implementing?

Ultra Champion

The Monitoring Console is sensational - just an example, we have 8 indexers and the two newest ones behaved in a very weird way through the MC in the past month or so. You look deeper and you find that the ulimit for file descriptors was low. Through these well thought interfaces, you can easily see what's going on - the queues that fill up and don't release...

Another example, a couple of weeks ago, we had a self inflicted denial of service attack at Where does the forwarder enqueue files?

Through the MC we identified the issue.

It's a must have component, from my point of view.

Highlighted

Re: Is the Monitoring Console (MC) tool really worth the effort of implementing?

Path Finder

Thanks for your honest opinion ddrillic. Good to hear that MC drills down for such things as ulimit. Sounds like a great tool.

0 Karma
Highlighted

Re: Is the Monitoring Console (MC) tool really worth the effort of implementing?

Ultra Champion

You are welcome - let's keep in mind that the ulimit issue was implied ; -)

0 Karma
Highlighted

Re: Is the Monitoring Console (MC) tool really worth the effort of implementing?

Splunk Employee
Splunk Employee

Concerns
I found the DMC tool (predecessor to MC) already present on scattered nodes in our environment.
1. How do I migrate from DMC tool to MC tool? Do I remove the existing DMC on nodes where found?

There is no fundamental difference between DMC and MC, only a slightly more accurate feature name. I've updated the Splexicon with information that I hope makes this a little clearer:

http://docs.splunk.com/Splexicon:Monitoringconsole

As far as seeing the tool on every node in your deployment -- yes, DMC/MC comes with Enterprise and by default is unconfigured and can see some information about its local instance (as described here:
http://docs.splunk.com/Documentation/Splunk/6.6.0/DMC/WhatcanDMCdo). To centrally monitor your entire deployment, follow the multi-instance deployment setup steps.

  1. Does MC actually run on V6.4.1 splunk enterprise or do I need to upgrade to 6.5 and above?

6.4.1 is fine. Upgrading to 6.5+ gets you the health check. Upgrading to 6.6+ gets you the data quality dashboard. As always, check the release notes for each version to learn about new features.