Getting Data In
Highlighted

Forwarder is listed in Deployment Monitor but not in Search App

Path Finder

We are running Splunk 4.3.3 (in the process of upgrading but we are stuck on this version for the moment), and one Windows system is not listed in the Search app in the Hosts list. I have searched all internal and non-internal indexes, and can't find any indexed data from this host. It does show in Deployment Monitor, and it shows that it is actively receiving data from this forwarder.

I have reinstalled the Universal Forwarder on this machine several times, but so far nothing changes. The GUID for this forwarder is not duplicated anywhere that I can find.

Any ideas?

Update [10/16/13]:
I compared metrics.log on a forwarder that is working and the one that is not. I noticed that the working system has entries with series="winevent..." (also entries for perfmon) and the system that is not working does not have winevent or perfmon anywhere in metrics.log. At this point, I'm thinking that splunkd can't read the Event Viewer logs or perfmon, but I'm not sure why yet, since splunkd is running as SYSTEM.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Super Champion

Have you verified that data is actually getting in?

Search sourcetype="*security*" ComputerName="nameOfTheComputer*"

If you get any results, check the host name for those events.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Path Finder

There is no data being indexed from this host. I have searched for the hostname in all indexes, and the only index that has that hostname or IP is _internal. I see events there where it appears data is being received from this host, and it shows as actively receiving data in the Deployment Monitor app. Events are being generated in Windows and are viewable in Event Viewer, and I have verified using netstat that there is an active connection to the receiving port on the Splunk indexer.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Super Champion

If everything is working, but you can't find the data, then it is most likely because it is stored with the wrong time (did you check the time on the host?), or it is a problem with the host name. The ComputerName is not the host name, it is a field in the Windows event logs.
If the bios battery is as old as your Splunk version (no offense) then the system time might have reset to 1970.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Path Finder

The host is brand new, and the time is correct (synced to NTP server).

Also, I ran the search you suggested, and it returns no data. The same search with a different Windows host does return data for that host. I checked outputs.conf and server.conf for this host and a working host, and they are essentially identical other than hostname and GUID.

As I said, data is being received from this host according to Deployment Monitor, but I can't figure out why it isn't indexing any of it.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Super Champion

Check the permissions on the inputs.conf (and everywhere else).
Is this WMI or monitor source data?
If it is a new system, have you tried re-installing the forwarder?

What Windows OS is this server running?

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Path Finder

Permissions on inputs.conf are identical on the Windows host that does have indexed data and the one where I can't find any indexed data. The Universal Forwarder is running under the Local System account, so there shouldn't be any permissions issues there.

The data that I am expecting to see is Windows Event Log data.

I mentioned in the original post that I have reinstalled the forwarder on this machine several times.

Hosts are running Windows 7 Ultimate x64 w/ SP1, and the Splunk indexer is running Windows Server 2008 R2 Enterprise w/ SP1.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Super Champion

How about the local and domain policies for access to the audit logs?
Windows 7 is sort of overboard on security, and the Splunk forwarder version is way behind that time. Is this the first Win7 system?
Local system permissions will be a problem if you're looking for WMI data.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Path Finder

The local policies appear to be identical, and the same domain policy is applied to all Windows 7 systems. There are no Group Policy errors shown on any of the systems, working or not working. This is not the first Windows 7 system; there are 3 other Windows 7 systems and 4 other Windows Server 2008 R2 systems that are all forwarding data to Splunk, and all but this one Windows 7 system has indexed data.

Splunk 4.3.3 was released in January 2012, so I don't suspect the age of the forwarder is an issue, especially since other Windows 7 and Windows Server 2008 R2 hosts are working fine.

0 Karma
Highlighted

Re: Forwarder is listed in Deployment Monitor but not in Search App

Super Champion

Well, you win the 'Splunk did something weird this week' award.
Have you turned debug on at the forwarder?

0 Karma