Deployment Architecture

What is the ideal DMC architecture for large environments?

milo
Observer

Hello all! I've inherited a large Splunk deployment and I've been given some leniency with setting up, or rather, revamping the monitoring.

Environment:
- 3 Primary locations
- 30 - 40 Indexers per location
- 10 - 12 Search Heads per location
- 1 DMC per location

** The numbers above don't account for BCP or lower environments.

Right now each DMC is responsible for its location, however there is a push to have the entire deployment's "health" available in a "single pane of glass".

Without regard to cost, what is the ideal method to accomplish this? I've toyed with standing up a single DMC at one of the regions and plugging all indexers, SH, etc. into it simply for the health perspective. The same scenario as I just mentioned but in Splunk Cloud is also possible. Also using one of the existing DMC's as the master for all regions is on the table.

I'd love to hear what is currently out there, or what architecture makes the most sense in this scenario.

Cheers!

 

 

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust
Consider installing the Mothership App for Splunk (https://splunkbase.splunk.com/app/4646/) on one of your MCs.
---
If this reply helps you, Karma would be appreciated.

ragedsparrow
SplunkTrust
SplunkTrust

The great thing about the Monitoring console (formerly DMC) is that you can group your environments with custom groups with Distributed mode, so you can get everything in your big pane of glass, but also do drill down by deployment and instance. 

So if you have, say 3 clusters, then you would simply label them (this could include search head clusters and such) in custom groups.  The dashboards are already set up to function with custom group labels, so there wouldn't be any real issue with just adding them to one well-sized Monitoring Console server and setting them up with the correct groups that are pertinent to your deployment.

0 Karma

milo
Observer

Thanks for the reply! 

Would you stand up another server to act solely as a "global" monitoring console and hook clusters from all locations into this new monitoring console? I'm thinking of a way to bring together all 3 "DMCs" into one, or rather what is the best practice / ideal way of doing this. 

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

How often do you plan on using this instance? Is this more of a land here occasionally during business hours and check cluster status? How many users will be doing this? That will effect sizing for sure.

Aside from multisite considerations { bandwidth from "master MC" to clusters } this should be easy to deploy and assuming this isn't a heavily used instance, a medium spec SH should be able to get you through your requirements. A VM would be sufficient for this. And of course, you can monitor your resources on that box.  And doing this from a one instance for everything is preferred, make your life easier (assuming you have all the above checked off.) 

Do be aware that most of the MC dashboards are REST driven so keep that in mind.

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...