How to solve error in monitoring console?

[Robertoing]

Hi to all,

I have three machines: 1 deployment-server, 1 SH/Indexer and 1 forwarder.
Looking at "monitoring console-panoramics" on deployment-server, i don't see the correct configuration (is available only deployment server, SH/Indexer and forwarder are not visible).

The data arrives correctly in the index and in "forwarder management" I see correctly the forwarder client.

Finally, the lookup "dmc_forwarder_assets" is empty.

Can someone help me please? Thanks. 

[VatsalJagani]

@Robertoing 

• Have you added all these machines for distributed search? - https://docs.splunk.com/Documentation/Splunk/8.2.6/DistSearch/Configuredistributedsearch 
• Have you done monitoring console-setup in distributed mode? - https://docs.splunk.com/Documentation/Splunk/8.2.6/DMC/Configureindistributedmode 
• Have you enabled forwarder monitoring? - https://docs.splunk.com/Documentation/Splunk/8.2.6/DMC/Configureforwardermonitoring 

If you perform this, MC should work as you expect.

----
I hope this helps!!!

[Robertoing]

Hi VatsalJagany,

maybe is not possibile configure the monitoring console on deployment-server (as distributed environment)  if search head and indexer are in the same host, because I tried to "Add new peer search" by Splunk web of the SH but I received error because the server names overlap.

It's possible or have you any idea?

Thanks you for the recent comment.

[isoutamo · Accepted solution]

Hi

If you have those (SH and IDX) as a separate roles on one splunk server then just add this once to MC as a remote peer. Then inside MC configuration give both roles to this node.

If those are separate splunk processes on same node then you must add those as individual servers with separate management ports (usually 8089 as a default). Any how this is not an recommended setup to run several splunk servers on as on separate processes on one server. You should use only one process on much better to use separate servers for those.

r. Ismo

[Robertoing]

Thank you isoutamo!

I have configured remote peer on Splunk Web of deployment-server instead on Splunk Web of SH/Indexer.

Lookup asset table is correctly valorized, but I see listed the Deployment server host; in "General configuration" I set unique server role as  Deployment-server, but I still see it in lookup asset table; it's correct?

[isoutamo]

When you have small environment max 50 (or something like that) UF/HF on your DS, you could use DS as a MC node. If you have lot of nodes then you need a separate DS and also I propose to use separate SH and IDX cluster with at least 2-3 peers and manager. Then you probably need a separate MC, don't put it into SH or individual search peer!

In those cases you should add SH/IDX node as a search peer (in DS) in distributed search GUI to it to able to query anything from SH/IDX logs. Also you should send your DS's logs to that node (as a best practices). 

Then you should configure correct roles for all those servers

• DS -> DS + MC (maybe SH+KVstore also)
• SH/IDX -> SH, IDX, KVstore
• add LM role to node which you are using as LM. Basically that could be you DS/MC or SH/IDX 

After that you can enable FWD monitoring on MC's Setting.

Now you should see those on correct groups/roles on MC.

[VatsalJagani]

For search peers, it's for distributed monitoring console. And that will require having different servernames. It seems currently your hostnames are conflicting.

You can change the name of the server like this - https://community.splunk.com/t5/Getting-Data-In/How-can-I-change-the-default-hostname-in-Splunk/m-p/...

./splunk set servername foo.domain.com
./splunk set default-hostname foo.domain.com

For Forwarder Management (dmc_forwarder_assets) you don't need that different hostname requirement. You can configure that anyways.

Please read this as well to define where to setup MC - https://docs.splunk.com/Documentation/Splunk/8.2.6/DMC/WheretohostDMC