Windows Events Not showing Up on Indexer

[pfabrizi]

A UF was installed on 2 Windows domain Controllers. These are in a different windows forest than my other devices. I had to manually add these to the windows_eventlog class by IP as the DNS name can't be resolved. I now see them sending to the indexer but I can't search any of the events. How can I trouble shoot this?

Thanks!

[gjanders]

You could read the metrics.log (documentation here) of the universal forwarder, the series= lines should advise which index, sourcetype and source the data was going through the forwarder.
The metrics.log/splunkd.log should confirm that the forwarder is forwarding as expected.

The tstats command might also help here, for example you could do:

| tstats count, max(_indextime) AS mostRecent, max(_time) AS mostRecentParsedTime where index=windows groupby host | eval mostRecent = strftime(mostRecent, "%+"), mostRecentParsedTime = strftime(mostRecentParsedTime, "%+")

You could then narrow down to an index/sourcetype/source or similar and include/exclude hosts until you narrow down to where your hosts are, perhaps they used IP instead of DNS name?
Since tstats queries metadata only it is quite quick to run over larger periods of time, however you can only use the where clause against indexed fields...

Finally, if you can see data leaving the forwarder but you are unsure where it's going run the btool command:

splunk btool outputs list --debug

[davebrooking]

Hi

How have you determined that the events are being sent to the indexer?

Could it be that the date format of the events is being misinterpreted and the events indexed today from the Domain Controllers are being indexed with a timestamp of 10 February 2017?

Dave

[pfabrizi]

I will check for that, It appears that I have to use the IP address and I had to manually add them to the server class on my deployment server. If I tried by DNS name the apps for the windows server class was not added, however when I added the IP they got configured. I have not had to do that with any other servers that I am aware of.