I'm receiving many errors (to the tune of 20GB/day from one server) in my _internal from a light forwarder.
Target: Windows 2k8 Splunk 4.1.5 running as local system Light Forwarder Desc: Splunk test forwarder. I am testing splunk as a log forwarder on windows, and this box is used for that purpose. No apps are actively running on the box (such as web servers etc) that would generate extra logs.
Indexer: RHEL 5 Splunk 4.1.3
Problem: In 15 minutes I receive 1,262,353 events from the Target server on my '_internal' database. 25% of these logs are "WinEventLogChannel - getBookMark: No checkpoint file available". Other errors that appear to occur significantly are "WinEventLogInputProcessor - main-thread: Failed to initialize Window Event Log 'various'" and "WiEventLogChannel - init: Init failed, unable to subscribe to Windows Event Log channel 'various'"
These errors sound like the Splunk instance is having trouble accessing certain windows logs. How do I turn these off, or better yet, grant access to Splunk to index them?
Splunk Light Forwarders will send internal logs in 4.1.x and above versions of Splunk. To disable them, you can follow the instructions here:
Additionally, you probably have a permissions problem with the user running Splunk on your Windows system. The user running Splunk should have service capability to access system level information.
I have been working with Splunk support, and we traced this down. Somehow I had gotten over 400 inputs added to my inputs.conf. Several of these events MS does not allow the logger to attach to and those were producing the errors. By removing the excess inputs, my processor and disk utilization dropped dramatically. The system is now reporting a usable amount of logs and working well.
Thanks. That provided me a way to stop my absurdly large log file. Any idea how to check the permissions? The user running Splunk is "Local System", I was pretty sure he had access to everything. I tried changing the splunk user to a different admin account that can view the log files in event viewer, but I still get the same spam errors.