DB Connect 2 health dashboard shows "no results found" on heavy forwarders in a distributed environment. Why?

[saikatr]

We have Splunk DB Connect 2 installed on two heavy forwarders, but the health dashboards work on none ("no results found"). We have another environment with DB Connect installed on the search head. In that case, the dashboard seems to work fine.
I know that the internal logs from the heavy forwarders are forwarded to our indexers. Is that the reason why the heath dashboard won't work?
Note: I also don't see any errors when I click the "RPC Service", (which also runs a search on the internal index).

I believe its standard practice to forward internal logs from HFs to indexer. In that case, what is the point of the health dashboard on a distributed environment? (This is of course assuming that my analysis of the problem is correct in the first place).

Splunk support were not able to suggest any solutions when we raised this as a case. They advised upgrading the app, which we did (from 2.0.3 to 2.1.1). This made no difference to the issue.

[MMCC]

Hi all,
answered Tomasz question in a seperate section. I'm not able to post links yet... not enough carma but here my explanation:

Are you running a distributed environment or a single one?

Does your outputs.conf send all the data to the indexer?
If yes, then there will be no local copy on the forwarder and thus the information is only available on the indexer.

To enable the forwarder to see the data you will have to add it as a "search head". If you only have one indexer you can do this via GUI in the Settings > Distributed search > Search peers. Add here the indexer to the forwarder.
If you have two indexers including the master node you will have to add both.
The easiest way is to execute this command (adding the forwarder as searchhead to the master):
$SPLUNK_HOME/bin/splunk edit cluster-config -mode searchhead -master_uri https://master_node_name:8089 -secret your_secret -auth admin:pwd

Make sure not to add the forwarder to the search head cluster! Otherwise the data will now be replicated here aswell!

I hope the solutions help you. Would like to know if it resolves your situation.

[Skorfulose]

Hello saikatr, tomaszwrona,

try configuring distributed search on the HF. Add your indexers (storing the dbx2 health logs) as search peers. Then the Health dashboard will work.

As you already pointed out if you forward your HF's logs to the indexer, they are not available on the HF. The dashboard won't find any result if distributed search is not configured.

Kind regards,
Thomas

[tomaszwrona]

hi saikatr,

i'm facing the same problem. Did you by any chance found a solution for this?

cheers
tomasz