Splunk Enterprise Security

Why did searches stop returning results after enabling the Distributed Management Console on a Splunk App for Enterprise Security search head?


After enabling the Distributed Management Console on an Enterprise Security (ES) search head, searches stop returning results. The following additional behaviors were also experienced.

  1. Navigation to the Settings menu in the UI is slow but works.
  2. When trying to access the Search UI, Splunk does not respond.
  3. Running a search from the command line does not return results.
  4. The “Scheduled time” value in “Settings > Searches, reports, and alerts” contains dates from the past.
  5. Large gaps in time (more than 24 hours in some cases) were seen in the scheduler.log and splunkd.log files.

Why is this happening?

0 Karma

Splunk Employee
Splunk Employee

Dont run DMC on the same search head as ES. There are some known issues with running DMC on a production SH, and it can effect search distribution.

0 Karma

Splunk Employee
Splunk Employee

The Distributed Management Console (DMC) makes changes to a number of different files. One of which is the addition of search groups in distsearch.conf which will change the way in which searches behave. The configuration documentation contains the following warning.

Important: Except for the case of a standalone, non-distributed Splunk Enterprise deployment, the instance hosting the DMC should not be used as a production search head and should not run any searches unrelated to its function as the DMC.

Also, when enabling the distributed mode of the feature, the user is presented with the following screen.

alt text

The only solution to correct this is to disable DMC on the search head. Instructions for doing this can be found in the documentation.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...