Why am I unable to add hosts with more than one Splunk instance to the Distributed Management Console?

[mfrost8]

Hi.

I'm wondering if I'm missing something here, but it seems like I can't manage a server that has more than one Splunk instance on it using the Distributed Management Console (Splunk 6.4+).

I noticed that I needed to have a different serverName set via server.conf in order to be able to add it as a search peer on my DMC. So same hostname, but different instance name and, of course, different management port. When I go to the DMC Setup page to turn on monitoring on my second instance and then hit Apply, I get

You have some unresolved errors that need to be fixed before you can proceed. Check the problems column and expand for more detail.

and then under the specific hosts (both entries with the same hostname but different instance names show this):

Duplicate instance name. Ensure each instance has a unique instance (host) name.

I don't see a way around this and it's kind of a bummer to not be able to see some of my deployment servers' status in the DMC. It's frustrating that I've done what was needed to make Splunk accept the second instance as a search peer, but the DMC wants more than that.

Thanks

[hexx · Accepted solution]

Duplicate instance name. Ensure each instance has a unique instance (host) name.

This message highlights one of the DMC requirements:

Make sure that each instance in the deployment (each search head, license master, and so on) has a unique server.conf serverName value and inputs.conf host value.

I would hope that the "Learn More" link should take you to a documentation topic that points out this requirement. If not, please let me know and I'll file a bug.

In essence, all instances monitored by the DMC need to have different values for both of the following properties that set the instance's name:

• server.conf / [general] / serverName
• inputs.conf / [default] / host

In your case, you need to edit etc/system/local/inputs.conf on instance "marigold-ds" and set the value of host in the [default] stanza to "marigold-ds", then restart this instance and run the DMC setup again.

[mfrost8]

Aha. Thanks. So then does it matter if the hostname listed in the second instance (marigold-ds in this example) is not a real hostname or alias? That is, Splunk will work with it just fine if it's just an arbitrary label that isn't resolvable to anything in DNS?

Thanks

[hexx]

does it matter if the hostname listed in the second instance (marigold-ds in this example) is not a real hostname or alias?

No, it doesn't matter. The settings we are talking about here represent an arbitrary label for your Splunk instance, which can be completely decorrelated with the hostname of the server that the instance runs on.

[mfrost8]

Great. This seems to have done it for me. I guess I missed this point about the hostnames in the documentation. It was made a little bit more confusing in that the definition/addition of search peers on the DMC then isn't as picky and can use the same DNS hostname with a different port.

In any case, I'm all set now. Thanks!

[hexx]

But despite the fact that that makes the search peer configuration happy, it's apparently not enough to make the DMC happy enough to add it. Or at least to let the DMC start monitoring it because the machine name is the same.

@mfrost8, this is unexpected. While the DMC requires for the instances it monitors to be uniquely identifiable based on the values of "host" (as defined in inputs.conf / host) and "splunk_server" (as defined in server.conf / serverName) associated with the events they read & return, "machine" (which represents the hostname of the server on which the Splunk instance is running) does not need to be unique.

While co-hosting Splunk instances is not something we necessarily recommend, it is supported to monitor co-hosted instances with the DMC.

Can you be more specific about the behavior you are seeing?

[mfrost8]

It was unexpected to me too :-).f

I have 3 servers this way -- with 2 instances, one using the normal 8089 mgmt port and one using 8189. All are Linux servers. Let's consider the server I'll call "marigold". Note that "marigold" is a more user-friendly DNS CNAME for the host that we use within Splunk rather than the regular hostname which is less pleasant -- we'll call that "mg1234.example.com". The $SPLUNK_HOME/etc/system/local/server.conf file has

...
[general]
serverName = marigold
...

The secondary instance on that same server (the one using port 8189 as a management port) has

...
[general]
serverName = marigold-ds
...

My first step in getting the DMC to recognize these instances was to add them as search peers on the DMC. I had no problem adding the "marigold" instance, but then discovered that it wouldn't take the second instance unless I set the serverName differently in server.conf above. After I did that, they both had an OK status in the search peers listing on the DMC.

If I then go to Settings->General Setup on the DMC I see both instances listed. At this point, I'd already successfully configured the "marigold" instance so it shows as configured. So I see

Instance (host)     Instance (serverName)    Machine   ...   Monitoring    State
mg1234                 marigold                        mg1234          Enabled         Configured
mg1234                 marigold-ds                  mg1234           Enabled         New

(on a side note, why is the Instance(host) column a large font size than the rest of the table?)

If I expand these I have the following for the first entry (remembering that marigold.example.com is a DNS CNAME for mg1234.example.com)

Peer URI    marigold.example.com:8089
OS              Linux
Cores         1
RAM           3964MB
Version      6.4.3

and

Peer URI    marigold.example.com:8189
OS              Linux
Cores         1
RAM           3964MB
Version      6.4.3

If I select the second entry (marigold-ds -- the one that's marked as "new") and hit the drop-down to the right to Edit Server Roles then change it to a deployment server, I get the pop-up that tells me this was done successfully. I then scroll up and click on Apply Changes. Now I get the Error pop-up with

You have some unresolved errors that need to be fixed before you can proceed. Check the problems column and expand for more detail.

The little red triangle exclamation marks are to the right of the marigold and marigold-ds entries. When I expand either of the two rows they both now show:

Duplicate instance name. Ensure each instance has a unique instance (host) name.
Resolve these problems to ensure that your dashboards are complete. Learn more

and that's about it. Thanks.

[hexx]

(on a side note, why is the Instance(host) column a large font size than the rest of the table?)

I think the idea there is to highlight / underscore the "primary" instance name that the DMC uses to identify instances. It should probably be "Instance (serverName)" that is highlighted, though, as it is that value that we use to populate the "instance" pull-downs. I'll file a bug.

[pradeepkumarg]

Did you try changing the instance name for one of the instances on that server?

server.conf

[general]

serverName =  yourHost_someidentifier_to_identify_the_instance

-The name used to identify this Splunk instance for features such as

  distributed search.

-Defaults to hostname

[mfrost8]

Yeah, as I mentioned I had to set a different instance name via server.conf in order to even add that second instance as a search peer. But despite the fact that that makes the search peer configuration happy, it's apparently not enough to make the DMC happy enough to add it. Or at least to let the DMC start monitoring it because the machine name is the same.

Thanks