All Apps and Add-ons
Highlighted

Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Path Finder

I'm using Splunk Light as an indexer, Splunk Heavy Forwarder for filtering and forwarding, and the Splunk Universal Forwarder on a Windows 2012 machine (not on the domain) to test Windows events.

I installed the Universal Forwarder on the Windows system and had it log everything, sending the data to the Heavy Forwarder. Except for having to install the Splunk Add-on for Microsoft Windows (which I think built the indexes?), everything seemed to work ok.

I uninstalled the Universal Forwarder and reinstalled it to capture only the Security log events, but they're not showing up.

I edited inputs.conf, for some reason it only has this by default:

[default]
host = CA3SYSBSCAN01

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

I added

[WinEventLog:Security]
disabled = 0

And restarted the agent, to no effect.

I've uninstalled, rebooted, reinstalled, stopped the agent, cleared the event logs, started the agent. It's not clear to me what's going wrong. This should be easy right?

Security event logs show up fine in the Event Viewer. I even installed nxlog as an experiment and it forwards events fine. (With its own set of problems... I would prefer the Splunk UF)

There must be a troubleshooting step I'm missing. How can I check the event volumes being observed and transmitted by the Universal Forwarder? How can I see if they're properly received on the Heavy Forwarder? That I have the right indexes installed on the Splunk Light indexer?

Thanks,

0 Karma
Highlighted

Re: Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Builder

To list out the indexes that you have defined:

./splunk cmd btool indexes list 

If you'd like to see Indexing volume per sourcetype you can search for the following:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart sum(kb) by series | sort - sum(kb)

Start with those two things and let us know where you're at. Also as an FYI if you were receiving data and you didn't have an index defined, Splunk would display a message in the GUI to let you know that.

Highlighted

Re: Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Path Finder

Very helpful, thanks. It looks like events are flowing into the system.

Searching on index="wineventlog" even shows the events. Searching on * doesn't.

Now my mystery is why the events aren't showing up in my searches.

0 Karma
Highlighted

Re: Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Motivator

check your user role and look into the indexes searched by default. if you don't specify the index splunk will only search on the default ones

------------
Hope I was able to help you. If so, an upvote would be appreciated.
Highlighted

Re: Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Builder

Agreed with what @diogofgm has stated. If the following search returns events

index=wineventlog

And this search does not:

index=*

The next place you'll want to look is at your roles. There you can limit access to specific indexes and set the index that are searched by default.

http://docs.splunk.com/Documentation/Splunk/6.4.1/Security/Aboutusersandroles

0 Karma
Highlighted

Re: Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

Influencer

Make sure you specify the destination index in inputs.conf as well, and restart the forwarder.

 [WinEventLog:Security]
 disabled = 0
 index = yourindex
0 Karma