Getting Data In
Highlighted

How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

Builder

Here's my setup: I have three clustered indexers, two search heads, a deployment server, as well as several Heavy Forwarders (three Windows and three Linux). I've been collecting Windows logs remotely from the HF via WMI no problems for a while. This week, I decided to install a universal forwarder on two servers as a pilot in preparation for further deployments.

After installing, I found I was getting no log events at all. So I commenced troubleshooting.

First I checked to see if the indexers were receiving data by running tcpdump and I saw the logs and metrics coming over the wire to the indexers. CHECK

Then I checked to see if the records were in ANY index by running the following search:

index = * host=hostnames

This returned nothing. So I searched:

index=* hostnames

And while this returned multiple events, none were FROM those machines.

Then, I checked to see if there were records in the _internal index from those servers. CHECK

Then, I looked to see if any of those _internal records contained errors. No entries that said ERROR, so tentative CHECK

Then I looked on each server where where the UF was installed and looked in splunkd.log for errors. Just one:

AuditTrailManager - Private key error Error opening C:\Program Files\SplunkUniversalForwarder\etc\auth\audit\private.pem: The system cannot find the patch specified.  

But I was kind of expecting this as I told the UF to use Splunk own internal certificate during install? Not sure if this is a factor....

So no other errors.

Here's C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkTAWindows\local\inputs.conf

[WinEventLog://Application]
disabled = 0
index = wineventlog

[WinEventLog://Security]
disabled = 0
index = wineventlog

[WinEventLog://System]
disabled = 0
index = wineventlog

[WinEventLog://Windows Powershell]
disabled = 0
index = wineventlog

Here's C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf

# BASE SETTINGS

[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = ip1:9997, ip2:9997, ip3:9997

## autolbsettings
autoLB = true
autoLBFrequency = 15
forceTimebasedAutoLB = true

Some other posts have mentioned that there could be a permissions issue. Is there a way to verify that? I installed this UF with the same domain admin account that the HF are using to pull logs via WMI so there shouldn't be a permissions issue?

What other steps can I take to fix this?

Thanks.

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

Builder
0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

SplunkTrust
SplunkTrust

To check permissions the account has...

runas /noprofile /env /netonly /user:domain\username "c:\windows\system32\eventvwr.msc"

you will be asked for a password.

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

Builder

So here are the results:

runas /noprofile /env /netonly /user:domain\username "c:\windows\system32\eventvwr.msc"

RUNAS ERROR: Unable to run - eventvwr.msc
193: eventvwr.msc is not a valid Win32 application.

To verify I ran just eventvwr.msc. That worked

I ran runas /noprofile /env /netonly /user:domain\username "notepad.exe"

That worked.

I tried both of the above from the command prompt AND the elevated command prompt with the exact same results.

Suggestions?

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

SplunkTrust
SplunkTrust

sorry, change .msc to .exe should work fine. since notepad.exe works fine, then removing c:\windows\system32\ should be ok too.

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

Builder

OK, that did work.

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

SplunkTrust
SplunkTrust

so as that user, can you read the logs?

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

Builder

Yes, I can read the logs.

0 Karma
Highlighted

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

SplunkTrust
SplunkTrust

well then the account has permission 😉

Are there a LOT of events in the logs? maybe from 2006 and beyond... if so it will take a while for the newer events to be read (depends on everything from size of the box to network throughput) etc. but events older than 6 years might be getting rolled to frozen as soon as they arrive, etc.

0 Karma