After enabling the Distributed Management Console on an Enterprise Security (ES) search head, searches stop returning results. The following additional behaviors were also experienced.
Why is this happening?
The Distributed Management Console (DMC) makes changes to a number of different files. One of which is the addition of search groups in distsearch.conf which will change the way in which searches behave. The configuration documentation contains the following warning.
Important: Except for the case of a standalone, non-distributed Splunk Enterprise deployment, the instance hosting the DMC should not be used as a production search head and should not run any searches unrelated to its function as the DMC.
Also, when enabling the distributed mode of the feature, the user is presented with the following screen.
The only solution to correct this is to disable DMC on the search head. Instructions for doing this can be found in the documentation.
Dont run DMC on the same search head as ES. There are some known issues with running DMC on a production SH, and it can effect search distribution.