Hi
CM (cluster master) is responsible to coordinate data replication, searches and distribute application bundles for cluster peers node. Also it could (should) use for indexer discovering (who source system UFs known where to send data) or other option is use fixed outputs.conf on UF e.g. via Deployment server.
Data is replicated based on SF (search factor) & RF (replication factor) on single site cluster. On multisite cluster there are also site_replication_factor and site_search_factor which define these over different sites.
More details can found here https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Basicclusterarchitecture.
CM tolds to SH (search heads) where those can found searches event by generation information.
What happens when your CM goes down? It depends how active your splunk environment is. Basically CM can be down for some time (hours), as basic feature in cluster don't need that CM is always available. When it's down all other nodes use their current states (e.g. generation list for searches, indexer location information for sending data, other peer information for replicate data). Depending on your cluster activities that time can be shorter or longer (hours) before there will start to be severe issues for it's work. The recommendation is that you should have stand by CM or ability to set it up quickly (within hours) if needed. Basically this means that you must have it's current configurations and apps in some safe place (e.g. your version management system). If/when needed you can build a new CM based on those with same name (I really hope/recommended that you use only fqdn not IP:s on your configurations!).
r. Ismo
