Splunk Add-on for Microsoft Windows: How to troubleshoot why Windows event logs are not being forwarded from the universal forwarder?

[mkallies]

I'm using Splunk Light as an indexer, Splunk Heavy Forwarder for filtering and forwarding, and the Splunk Universal Forwarder on a Windows 2012 machine (not on the domain) to test Windows events.

I installed the Universal Forwarder on the Windows system and had it log everything, sending the data to the Heavy Forwarder. Except for having to install the Splunk Add-on for Microsoft Windows (which I think built the indexes?), everything seemed to work ok.

I uninstalled the Universal Forwarder and reinstalled it to capture only the Security log events, but they're not showing up.

I edited inputs.conf, for some reason it only has this by default:

[default]
host = CA3SYSBSCAN01

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

I added

[WinEventLog:Security]
disabled = 0

And restarted the agent, to no effect.

I've uninstalled, rebooted, reinstalled, stopped the agent, cleared the event logs, started the agent. It's not clear to me what's going wrong. This should be easy right?

Security event logs show up fine in the Event Viewer. I even installed nxlog as an experiment and it forwards events fine. (With its own set of problems... I would prefer the Splunk UF)

There must be a troubleshooting step I'm missing. How can I check the event volumes being observed and transmitted by the Universal Forwarder? How can I see if they're properly received on the Heavy Forwarder? That I have the right indexes installed on the Splunk Light indexer?

Thanks,

[masonmorales]

Make sure you specify the destination index in inputs.conf as well, and restart the forwarder.

 [WinEventLog:Security]
 disabled = 0
 index = yourindex

[ryanoconnor]

To list out the indexes that you have defined:

./splunk cmd btool indexes list 

If you'd like to see Indexing volume per sourcetype you can search for the following:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart sum(kb) by series | sort - sum(kb)

Start with those two things and let us know where you're at. Also as an FYI if you were receiving data and you didn't have an index defined, Splunk would display a message in the GUI to let you know that.

[mkallies]

Very helpful, thanks. It looks like events are flowing into the system.

Searching on index="wineventlog" even shows the events. Searching on * doesn't.

Now my mystery is why the events aren't showing up in my searches.

[diogofgm]

check your user role and look into the indexes searched by default. if you don't specify the index splunk will only search on the default ones

[ryanoconnor]

Agreed with what @diogofgm has stated. If the following search returns events

index=wineventlog

And this search does not:

index=*

The next place you'll want to look is at your roles. There you can limit access to specific indexes and set the index that are searched by default.

http://docs.splunk.com/Documentation/Splunk/6.4.1/Security/Aboutusersandroles