Some Local Windows Eventlogs not being indexed

[marcpatron]

I am trying to index the local windows eventlogs, but there appears to be an issue reading the "Security" eventlog, and is then no longer indexing all the logs ongoing. On restart of splunk the logs are being processed alphabetically, with a Processing event then a Finished event. It appears the Security log gets a Processing event, but not a Finished event.

I have cleared the Security Log (and other logs aswell), but the issue persists.

Has anyone else seen this issue?

\var\log\splunk\splunkd.log - Splunk 4.3.2 on Windows

10-31-2012 12:19:20.240 +1100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Security'

10-31-2012 12:18:59.194 +1100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'Internet Explorer': total_events='0' with empty_msg='0'.

10-31-2012 12:18:59.194 +1100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Internet Explorer'

10-31-2012 12:18:59.194 +1100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'HardwareEvents': total_events='0' with empty_msg='0'.

10-31-2012 12:18:59.194 +1100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'HardwareEvents'

10-31-2012 12:18:59.194 +1100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'ForwardedEvents': total_events='249' with empty_msg='0'.

10-31-2012 12:18:58.367 +1100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'ForwardedEvents'

10-31-2012 12:18:58.367 +1100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'Application': total_events='0' with empty_msg='0'.

10-31-2012 12:18:58.367 +1100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Application'

[marcpatron · Accepted solution]

The problem has been solved.

At the same time of a bunch of other changes, some firewall rules were put in place around the Splunk server. The WinEventLog:Security input by default looks up AD to resolve SID's in events (evt_resolve_ad_obj = 1). This uses RPC ports to communicate to the AD servers. I have disabled this setting (evt_resolve_ad_obj = 0) and all event logs are now being indexed once again. There appears to be no issue with resolved usernames in the eventlogs.

I discovered this in the splunkd.log with DEBUG turned on for WinEventLog*. Initially the following entries appear just as the Security log was begining to be processed:

WinEventLogChannel - EvtDC::bind: Found DC='\SERVER1.xyz.loc', DCsite='XYZ', ClientSite = 'XYZ', Domain='xyz.loc'
WinEventLogChannel - connectToDC: DsBind failed: (1722)'The operation completed successfully.'
WinEventLogChannel - init: Failed to bind to DC, dc_bind_time=21140 msec

Then every 21 seconds:

WinEventLogChannel - connectToDC: DsBind failed: (1722)'The operation completed successfully.'
WinEventLogChannel - WinEventLogChannel::translateSidLocally Translating sids locally...

I assume that the events were being indexed, just very slowly, so that it would appear to never finish indexing the security log and move onto other logs.

I have reviewed the firewall rules and need to allow the blocked RPC port (tcp/1026).

[splunkIT]

Also see this Answers thread:

http://splunk-base.splunk.com/answers/75970/forwarder-shows-extreme-lag-or-latency-when-sending-wind...

[marcpatron]

I am indexing using Local Event Log collection, configured in the Windows App, not via monitoring the .evtx files. The server is Win2008.

[rovechkin_splun]

can you please clarify your scenario? Are you indexing evtx logs by pointing Splunk to the directory?