[Multiple Servers with Same Hostname] Deployment Server Integration

[AhmadKhattak20]

Hi All,

I have a scenario where there are many servers having the same hostname due to some requirements of the applications running on them.

Now, the splunk universal forwarder agent has been successfully deployed on all of them and the inputs.conf and outputs.conf have been manually configured there. These are windows servers. 

It's very difficult to manage all of these servers manually by editing the inputs.conf so what I'm trying to do is manage them centrally via the Deployment Server. However after the deploymentclient.conf file has been configured there, all of the servers are not showing up on the Deployment Server because of them having the same hostname. I get one entry against the hostname on the Deployment Server.

My question here is that what changes do I need to make so that all of them report successfully on the Deployment Server? I've been thinking of pushing a deploymentclient.conf file via the Deployment Server with the clientName value set to $HOSTNAME-$IPADDRESS. Is this possible? What other environment variable can I use other than $HOSTNAME to make the clientName unique?

Lastly, when the logs are being received in Splunk, the host value that shows up there has been manually set for each server in the inputs.conf file with HOSTNAME-IP Address, so when I remove the manual configurations and push the inputs.conf via deployment server, will the host = $HOSTNAME-$IPADDRESS work?

Thank you. 

[s2_splunk]

The clientName attribute in deploymentclient.conf is used solely for the purposes of matching a deployment client to a serverclass entry. You can name that whatever you like to distinguish groups of hosts instead of relying on the hostname. 

To distinguish the events during search, your approach of explicitly setting the host to a unique value as you described will work.

[AhmadKhattak20]

Thank you @s2_splunk for your response.

So, then for multiple servers having same hostname and therefore not showing up on the deployment server, I can use the deploymentclient.conf - clientName attribute to be set for each server with unique value so that all of them show up on the Deployment Server.

Can you share any environment variables I can set in the clientName field so that every value is unique? As you know hostname is same for every server so is there any other environment variable or splunk value such as the one stored in instance.cfg that I can use here? Thank you.

[s2_splunk]

I am not sure I understand what you mean by "not showing up on the deployment server". If you do not set clientName in deploymentclient.conf, the deployment client(s) should show up on the DS with its/their GUID value as the Client Name.

I am not sure which environment variables are available and will be properly substituted by the deployment client code when processing the .conf file.  You should try to ensure the clientName ends up being unique; otherwise some of the forwarder management dashboards on the deployment server will not display accurate information (like distinct counts, etc.)

[AhmadKhattak20]

To clarify "Not showing up on the deployment server", it's that since all the servers have the same hostname so on the Deployment Server instead of having all of the e.g. 30 Servers show up I only get one entry shown with the hostname where the IP Address periodically changes/updates within the range of the IP Addresses assigned to the 30 Servers.

[AhmadKhattak20]

I just checked again now and by default it seems the clientName value is unique however since the hostname/instance name are same for the 30 Servers so that's why it's still showing up as one entry on the deployment server.

Any suggestions on how to deal with this scenario? Keeping in mind that the team cannot change the hostnames for the 30 servers to something unique rather all of them need to have the same hostname. 

Thank you.

[s2_splunk · Accepted solution]

I don't think there is an easy solution to fix that other than ensuring that each forwarder checks in with a unique hostname. 

The only way I can think of is to have your Linux admins ensure that an environment variable containing the current IP address is set on server startup (or add that to your splunk startup script). Then you can configure a unique serverName in server.conf, like so: 

[general]
serverName = <currentHostName>--$IP_ADDR

If your hostname is currently hostXYZ and your environment variable IP_ADDR is set to 192.168.5.5 (for example), you should see a distinct entry in Forwarder Management with the Instance Name field set to hostXYZ--192.168.5.5.

It will not change the hostname (nor should it), but it should address your issue.