Forwarder is listed in Deployment Monitor but not in Search App

[smwirt]

We are running Splunk 4.3.3 (in the process of upgrading but we are stuck on this version for the moment), and one Windows system is not listed in the Search app in the Hosts list. I have searched all internal and non-internal indexes, and can't find any indexed data from this host. It does show in Deployment Monitor, and it shows that it is actively receiving data from this forwarder.

I have reinstalled the Universal Forwarder on this machine several times, but so far nothing changes. The GUID for this forwarder is not duplicated anywhere that I can find.

Any ideas?

Update [10/16/13]:
I compared metrics.log on a forwarder that is working and the one that is not. I noticed that the working system has entries with series="winevent..." (also entries for perfmon) and the system that is not working does not have winevent or perfmon anywhere in metrics.log. At this point, I'm thinking that splunkd can't read the Event Viewer logs or perfmon, but I'm not sure why yet, since splunkd is running as SYSTEM.

[smwirt]

Issue resolved!

My apps were not deploying to this system for some reason, so I checked Manager > Deployment > Deployment Server. Per our customer's specification, deployment whitelists were used. This one system had a hostname that did not match the naming convention specified in the whitelist. After adding this system's naming convention to the whitelist, reinstalling the forwarder resolved the issue.

[lukejadamec]

I knew it had to be something simple.

[smwirt]

I enabled debugging and haven't found any issues so far in splunkd.log, but I did notice that on the problem system, metrics.log isn't showing any entries with series="winevent" or series="perfmon" (see my edit of the original question). splunkd is running under SYSTEM, so I wouldn't expect there to be permissions issues reading the event log or perfmon.

[lukejadamec]

-Navigate to $SPLUNK_HOME/bin.
-Stop Splunk, if it is running.
-Save your existing splunkd.log file by moving it to a new filename, like splunkd.log.old.
-Restart Splunk in debug mode with splunk start --debug.
-When you notice the problem, stop Splunk.
-Move the new splunkd.log file elsewhere and restore your old one.
-Stop or restart Splunk normally (without the --debug flag) to disable debug logging.
http://docs.splunk.com/Documentation/Splunk/4.3.3/Troubleshooting/Enabledebuglogging

You can also edit individual processes from this file:
$SPLUNK_HOME/etc/log.cfg

[smwirt]

I have not. I'm not familiar with how to do that, but I should be able to find out easily enough. I am out for the rest of the day, and will try this first thing tomorrow.

[lukejadamec]

Well, you win the 'Splunk did something weird this week' award.
Have you turned debug on at the forwarder?

[smwirt]

The local policies appear to be identical, and the same domain policy is applied to all Windows 7 systems. There are no Group Policy errors shown on any of the systems, working or not working. This is not the first Windows 7 system; there are 3 other Windows 7 systems and 4 other Windows Server 2008 R2 systems that are all forwarding data to Splunk, and all but this one Windows 7 system has indexed data.

Splunk 4.3.3 was released in January 2012, so I don't suspect the age of the forwarder is an issue, especially since other Windows 7 and Windows Server 2008 R2 hosts are working fine.

[lukejadamec]

How about the local and domain policies for access to the audit logs?
Windows 7 is sort of overboard on security, and the Splunk forwarder version is way behind that time. Is this the first Win7 system?
Local system permissions will be a problem if you're looking for WMI data.

[smwirt]

Permissions on inputs.conf are identical on the Windows host that does have indexed data and the one where I can't find any indexed data. The Universal Forwarder is running under the Local System account, so there shouldn't be any permissions issues there.

The data that I am expecting to see is Windows Event Log data.

I mentioned in the original post that I have reinstalled the forwarder on this machine several times.

Hosts are running Windows 7 Ultimate x64 w/ SP1, and the Splunk indexer is running Windows Server 2008 R2 Enterprise w/ SP1.

[lukejadamec]

Check the permissions on the inputs.conf (and everywhere else).
Is this WMI or monitor source data?
If it is a new system, have you tried re-installing the forwarder?

What Windows OS is this server running?

[smwirt]

The host is brand new, and the time is correct (synced to NTP server).

Also, I ran the search you suggested, and it returns no data. The same search with a different Windows host does return data for that host. I checked outputs.conf and server.conf for this host and a working host, and they are essentially identical other than hostname and GUID.

As I said, data is being received from this host according to Deployment Monitor, but I can't figure out why it isn't indexing any of it.

[lukejadamec]

If everything is working, but you can't find the data, then it is most likely because it is stored with the wrong time (did you check the time on the host?), or it is a problem with the host name. The ComputerName is not the host name, it is a field in the Windows event logs.
If the bios battery is as old as your Splunk version (no offense) then the system time might have reset to 1970.

[smwirt]

There is no data being indexed from this host. I have searched for the hostname in all indexes, and the only index that has that hostname or IP is _internal. I see events there where it appears data is being received from this host, and it shows as actively receiving data in the Deployment Monitor app. Events are being generated in Windows and are viewable in Event Viewer, and I have verified using netstat that there is an active connection to the receiving port on the Splunk indexer.

[lukejadamec]

Have you verified that data is actually getting in?

Search sourcetype="*security*" ComputerName="nameOfTheComputer*"

If you get any results, check the host name for those events.