Renamed Machine (VM Image) still sending Old Hostname

[jchensor]

I have a machine image created in our VMWare environment that was created by the IT department of my company. It was created off of their template images and then renamed to a new name. However, the template image already has Splunk installed on it before they rename it to the new Host Name. Unfortunately, after they've renamed the image, the events forwarded by Splunk to our Splunk environment are still using the OLD Host Name from the template as opposed to the new name the machines have been changed to.

Now, I went through a bunch of the conf files. I found two offending files in ".../etc/system/local":

1. inputs.conf - This had the old Host Name, I just deleted the file as that was the only contents of that conf file.
2. server.conf - This had a "ServerName" field with the old Host Name, and I edited to reflect the new Host Name.

However, after doing this, Splunk is STILL sending the old Host Name! Does anyone have any idea where else I may need to edit any conf files or something? Where else would the Host Name be stored? Has anyone else seen something similar?

EDIT: The plot thickens!!

I have two apps on the machine: one that forwards Windows Event Logs and another that forwards some custom logs on the machine. Interestingly enough, the app that forwards the Windows Event Logs are actually sending the correct new Host Name! However, the app that is forwarding the custom logs are still forward the OLD Host Name!

So one app works and the other doesn't. How can this possibly be? I've double checked the conf files for the app that isn't working and there is no reference to the old Host Name anywhere. Is somehow the app just holding on to legacy settings somehow? I've already restarted the Splunk service a few times.

Thanks!

• James

[jgbricker]

Adding my example in the hope that it helps someone else. The server administrator team recently added Splunk to their server build template. This template is used to build new servers. The template hostname was saved to C:\"Program Files"\SplunkUniversalForwarder\etc\system\local\inputs.conf under the [default] stanza. I removed that file on one of the new servers and restarted Splunk. I confirmed this corrected the host name issue. I then asked our server administrator team to remove that file (inputs.conf) from the ..\system\local\ path on the template. Now when a new server is generated using that template it will not put that file on it.

[bob_kerns]

A complication (and my guess as to what bit you) is that the hostname listed in the Deployment Monitor is the value of the HOSTNAME environment variable when you launch splunk.

So if you change the hostname, edit the inputs.conf and server.conf and remove the guid from etc/instance.cfg, and restart from the same session, likely your HOSTNAME environment variable will hold the old hostname.

This confused the hell out of me for a while, with it sometimes working and sometimes not.

But I verified: cleared the guid= setting, then did:

HOSTNAME=fred.flintstone.com splunk restart

Now fred.flintstone.com showed up in the Deployment Monitor, even though that appeared nowhere else but in the HOSTNAME environment variable.

So log out and log back in, or set your HOSTNAME variable environment manually, before restarting.

Full procedure:

• Set the desired hostname (hostname myhost.example.com)
• set host = in /etc/system/local/inputs.conf
• set serverName = in /etc/system/local/server.conf
• remove guid= in /etc/instance.cfg
• ensure you have the correct HOSTNAME environment variable, fix if not, e.g.: export HOSTNAME="$(hostname)"
• restart splunk
• remove old bogus entry from the Deployment Monitor page

i hope that helps the next guy!

[jchensor · Accepted solution]

Interestingly enough, it seems like it was just something that I had to... give time? The names have started being reported properly today, and I didn't change anything. Maybe it was just some sort of cached DNS entry or something. Whatever the case is, it's working now. shrug Wish I had a more solid reason why this happened.

[hexx]

Could you paste the output of splunk cmd btool inputs list --debug ?