topic Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs? in Getting Data In

How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Tue, 29 Sep 2020 09:36:03 GMT

Here's my setup: I have three clustered indexers, two search heads, a deployment server, as well as several Heavy Forwarders (three Windows and three Linux). I've been collecting Windows logs remotely from the HF via WMI no problems for a while. This week, I decided to install a universal forwarder on two servers as a pilot in preparation for further deployments.

After installing, I found I was getting no log events at all. So I commenced troubleshooting.

First I checked to see if the indexers were receiving data by running tcpdump and I saw the logs and metrics coming over the wire to the indexers. CHECK

Then I checked to see if the records were in ANY index by running the following search:

index = * host=hostnames

This returned nothing. So I searched:

index=* hostnames

And while this returned multiple events, none were FROM those machines.

Then, I checked to see if there were records in the _internal index from those servers. CHECK

Then, I looked to see if any of those _internal records contained errors. No entries that said ERROR, so tentative CHECK

Then I looked on each server where where the UF was installed and looked in splunkd.log for errors. Just one:

AuditTrailManager - Private key error Error opening C:\Program Files\SplunkUniversalForwarder\etc\auth\audit\private.pem: The system cannot find the patch specified.

But I was kind of expecting this as I told the UF to use Splunk own internal certificate during install? Not sure if this is a factor....

So no other errors.

Here's C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_Windows\local\inputs.conf

[WinEventLog://Application]
disabled = 0
index = wineventlog

[WinEventLog://Security]
disabled = 0
index = wineventlog

[WinEventLog://System]
disabled = 0
index = wineventlog

[WinEventLog://Windows Powershell]
disabled = 0
index = wineventlog

Here's C:\Program Files\SplunkUniversalForwarder\etc\system\local\outputs.conf

# BASE SETTINGS

[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = ip1:9997, ip2:9997, ip3:9997

## autolbsettings
autoLB = true
autoLBFrequency = 15
forceTimebasedAutoLB = true

Some other posts have mentioned that there could be a permissions issue. Is there a way to verify that? I installed this UF with the same domain admin account that the HF are using to pull logs via WMI so there shouldn't be a permissions issue?

What other steps can I take to fix this?

Thanks.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 14:22:11 GMT

I should say I've been looking at the following posts, but have not gotten a solution from them yet:

https://answers.splunk.com/answers/310305/after-installing-the-universal-forwarder-on-30-win.html

https://answers.splunk.com/answers/223604/why-is-no-data-being-indexed-after-deploying-the-s.html?utm_source=typeahead&utm_medium=newquestion&utm_campaign=no_votes_sort_relev

https://answers.splunk.com/answers/389351/why-is-my-universal-forwarder-on-windows-server-20.html

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 14:40:57 GMT

To check permissions the account has...

runas /noprofile /env /netonly /user:domain\username "c:\windows\system32\eventvwr.msc"

you will be asked for a password.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 14:54:08 GMT

So here are the results:

runas /noprofile /env /netonly /user:domain\username "c:\windows\system32\eventvwr.msc"

RUNAS ERROR: Unable to run - eventvwr.msc
193: eventvwr.msc is not a valid Win32 application.

To verify I ran just eventvwr.msc. That worked

I ran runas /noprofile /env /netonly /user:domain\username "notepad.exe"

That worked.

I tried both of the above from the command prompt AND the elevated command prompt with the exact same results.

Suggestions?

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 14:56:32 GMT

sorry, change .msc to .exe should work fine. since notepad.exe works fine, then removing c:\windows\system32\ should be ok too.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 14:58:48 GMT

OK, that did work.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 15:02:11 GMT

as follow up... adding permission for users can be tricky:

http://techibee.com/sysadmins/how-to-grant-permissions-to-view-security-event-log-in-windows-server-2003-and-2008/2116

https://msdn.microsoft.com/en-us/library/windows/desktop/aa363658(v=vs.85).aspx

https://blogs.technet.microsoft.com/janelewis/2010/04/30/giving-non-administrators-permission-to-read-event-logs-windows-2003-and-windows-2008/

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 15:05:58 GMT

so as that user, can you read the logs?

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 15:07:41 GMT

Yes, I can read the logs.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 15:16:15 GMT

well then the account has permission 😉

Are there a LOT of events in the logs? maybe from 2006 and beyond... if so it will take a while for the newer events to be read (depends on everything from size of the box to network throughput) etc. but events older than 6 years might be getting rolled to frozen as soon as they arrive, etc.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 15:22:10 GMT

The server is three years old and yes I forgot to put that limit into the .conf files.

Is there a way to determine where those events are going? If they are in any index?

While I had turned off the UF last night, it's been running now for four hours today and still nothing is showing up (I just checked)

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 15:32:57 GMT

I think there's been enough time by now.

You did create an index called wineventlog correct?

no typos etc.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 15:53:46 GMT

Just verified that it's spelled correctly in both inputs.conf

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 16:01:32 GMT

very interesting... can you check to be sure theyre not ending up in index=_internal

 index=_internal source="WinEventLog:*"

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 16:13:18 GMT

is it in internal?

index=_internal source="WinEventLog:*"

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 16:19:22 GMT

OK I checked _internal and no entries. So then I re-ran the search for anything from that server, just to see if anything turned up (index=wineventlog host=hostmanes) and lo and behold ONE event showed up!

But only one and from the system log.

So I expanded the search to all time (because why not) and it seems that your previous theory was right, I have VERY few events after midnight 3 May 2016, hundreds of thousands of events between 2pm and midnight and then almost nothing prior to that. 2pm on 3 May was about when I installed the UF.

So that leads to another question, can I stop the UF, add in the history limit of 3 days and restart? Or at this point will it ignore that config?

And why isn't it getting any logs after midnight?

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

jkat54 — Wed, 04 May 2016 17:02:18 GMT

I think i've hit a limit on comments because it keeps discarding my latest comments

are the events in the internal index?

 index=_internal source="WinEventLog*"

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 17:10:01 GMT

I didn't notice that my reply to your comment didn't get posted.

The events are NOT in the internal log

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 17:12:48 GMT

The events are not in the _internal log.

Furthermore, I performed a general search index=* host=hostname and found that I HAD gotten some results.

From 2pm 3 May 2016 to midnight 3 May 2016, I received about 100,000+ events per hour. Then it has dropped off to maybe one event per hour.

and even then, it's only been the events from the system log.

Re: How to troubleshoot why a Windows universal forwarder is sending metrics, but not Windows event logs?

reswob4 — Wed, 04 May 2016 17:35:49 GMT

I just tried the SPL99687 suggestion from http://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/KnownIssues

and when I stopped and restarted splunk, THOSE TWO log entries showed in the search. But still nothing else.

double checking spelling again....