Why does my Splunk 6.3.2 Distributed Management Console display incomplete and odd information?

[lycollicott]

Last week I setup a dedicated 6.3.2 DMC per the magic documentation, but it doesn't seem to be working correctly.

I labeled my clusters, but after I registered them as search peers the search heads had both labels as indicated below. There is also no role listed for my heavy forwarder

My Overview looks like this and seems to be aware of my indexes, but the Indexing pages do not return any results when I try to dig into the indexing details.

[ykou_splunk · Accepted solution]

I assume what you are asking is, why the indexers don't show up.

1. make sure your DMC instance is a search head of the indexer cluster. Go to Settings -> Indexer clustering -> enable clustering, then choose mode as "search head" and point to the cluster master, then restart.
2. go to Settings -> Distributed search -> Search peers, verify that the indexers are listed as peers on this page.
3. go to DMC setup page, now the indexers should be listed here, make sure they have the "indexer" server role. Then click "Apply Changes" button. Note clicking the "Apply Changes" button is important!
4. Now the overview page should show a new panel for the indexers and the indexing related dashboards should work.

NOTE: since your deployment is indexer cluster + search head cluster, please make sure the DMC is a search head in the indexer cluster while NOT being a search head cluster member. In another word, the DMC needs to be inside indexer cluster, and outside of search head cluster.

[lycollicott]

This doesn't seem to match the documentation, but I did step 1 like you suggested and then the indexers did not appear in either step 2 or 3.

The documentation says:
"Do not add clustered indexers, but you must add clustered search heads. If you are monitoring an indexer cluster, and you are hosting the DMC on an instance other than the cluster master, you must add the cluster master as a search peer."

[ykou_splunk]

can you log in the cluster master, then click Settings -> Indexer clustering, then clicks on the "search head" tab to verify that the DMC instance is listed in the table? In another word, please verify that the cluster master is aware of the DMC instance.

Also, can you login the DMC instance, then click Settings -> Indexer clustering, the page should show that this instance is a search head in an indexer cluster.

[lycollicott]

The master does not have the DMC listed as a search head.

On the DMC "Indexer Clustering page there is this error: "Master has multisite enabled but the search head is missing the 'multisite' attribute"

[lycollicott]

I manually added a [clustering] stanza to server.conf on the DMC with multisite=true then restarted, but the error persists.

[ykou_splunk]

Here's documentation for configuring search peer in multisite environment: http://docs.splunk.com/Documentation/Splunk/6.2.0/Indexer/Multisiteconffile#Configure_the_search_hea...

basically you need to add something like this:

[general]
site = site1

note the [general] stanza might already exists so you can just add site= to it

[lycollicott]

Yup, I was already headed in that direction. 🙂 Ok, so now the DMC shows up as one of the master indexer's search heads.

However I still see no indexers listed in the DMC pages and this error occurs: " Search filters specified using splunk_server/splunk_server_group do not match any search peer." That looks like SPL-99116, but that was for an older version not 6.3.2.

[ykou_splunk]

the reason is, DMC doesn't know what happened outside of it. As I described above, when you finish step 1 and step 2, (also configure multisite in your case), you need to go through step 3 which essentially re-configure DMC.

In another word, every time you change splunk environment configuration (for example, add/remove any splunk instance), you need to also go to DMC setup page and click "Apply Changes" button to let DMC know that the environment changed.

[lycollicott]

Hmm. I followed ii_splunk's suggestion in https://answers.splunk.com/answers/208043/unable-to-run-any-search-query-warn-search-filters.html and that seemed to somehow work. He said to do the following and boom it showed the actual cluster peer indexer. I have no idea why it worked though.

Settings->Distributed Management
Console (NOTE: Indexers will have N/A
shown) Setup->Apply Changes->Refresh
(NOTE: No changes were actually made)

Verify fix by clicking "Overview" in
Distributed Management Console;
Indexers will now show correct
indexing rate.

Search as normal; workaround complete.

[hexx]

Additionally, do note that the DMC does not currently have the capacity to directly monitor forwarders - this means that you should not attempt to set up any forwarder (heavy-weight or otherwise) as a search peer of the DMC for direct monitoring.

The DMC does offer the ability to monitor your forwarders, but it does so indirectly by querying logs recording incoming forwarder traffic by the indexers.

[lycollicott]

Thanks. I'll remove my HFs from the equation.