Our splunk has the windows app installed and we look at the WMI:LocalProcesses source for process monitoring. We have an observation here that the "Host" value showed at splunk search (eg. host=L33904 | sourcetype=WMI:LocalProcesses | source=WMI:LocalProcesses) does not match the actual computer name for that machine.
We have two incidents so far: 1. A machine with Host value "l3win7clone" (the computer name used in the pc clone) generates a huge amount of data to splunk. We want to locate the pc yet can't find "l3win7clone" at the DHCP server. According to the tech support personnel, the computer name should have been changed to its assigned name after the cloning. But it seems the reading from wmi is still using the original computer name. We can't ping or traceroute to this l3win7clone at all. Is there any good way to locate this machine, other than using the Host value? 2. I renamed a machine from X to Y. Yet currently the wmi reading at splunk still shows X, though the computer name at the machine is already changed to Y.
The discrepancy here gives some trouble to our system administration to map and locate these machines. Can someone advise how to address the issue? Is it a problem with splunk or the wmi source?
I've figured out what went wrong. "l3win7clone" is used in the master clone copy. After cloning, the machine name is changed but not the server name used in the splunk config file. hence, for splunk, it still uses "l3win7clone" as the host name. The huge amount of data are actually generated by many different machines that share the same server name.
The host value gets set by the server.conf configuration file, this is located at %SPLUNK_HOME%\etc\system\local or \etc\system\default if one has not been made in local. If this config file was messed up then it would give you problems with the host value. It seems weird that this would happen on only one machine if you are ghosting/cloning machines.
I assume you are using some sort of forwarder on your machines being cloned?
To locate which machine is being weird, I would suggest taking a look at the metrics.log file located on the Splunk Indexer at %SPLUNK_HOME%\var\log\splunk. Inside this log file there should be a "group=tcpin_connections" INFO line for each machine that is talking to the indexer. I would search for "l3win7clone" which should return a result for the hostname field on one of the INFO lines, i.e. "hostname=l3win7clone". At the beginging of this line should by an IPaddress field. You should be able to find the server once you have the ip address.
Hope this helps!
hi proctorgeorge, yes, we forward the logs from different machines (including the l3win7clone) to a splunk "server". I've looked at the metrics.log at the splunk server. However, the hostnames there are using IP instead of "hostname". Is there sth wrong with the config we used?
I also tried using netstat to try matching the IPs and ports there. But it does not seem to help.