Security Windows Event Logs not collected by the UF on Windows 2008 and Windows 2012 - Splunk 6

[mgaraventa_splu]

Hi all, I need your help regarding SDDL (Security Descriptor Definition Language) configurations for setting the right channel security descriptor for Security Events on Windows. My customer is monitoring successfully Windows application and system logs using UFs on DCs which send those events to the main IX. For 2 DCs (one Windows 2003 and the other Windows 2012) the UF is collecting also Security logs, but with no success: they apparently do not arrive to the IX. All Windows Event Log inputs (Application, Security and System) are configured in the same way and I cannot spot any misconfigurations. The btool output looks correct as well.

For some reason I cannot find in the splunkd.log any WinEventLogInputProcessor entries at all (I would have expected to find at least some for the Application and System logs, like the ones in the example below):

02-03-2014 12:13:29.360 +0100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'System': total_events='197' with empty_msg='0'. 
02-03-2014 12:28:31.048 +0100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Application' 
02-03-2014 12:28:31.048 +0100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'Application': total_events='83' with empty_msg='0'. 
02-03-2014 12:28:31.407 +0100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'System' 
02-03-2014 12:28:31.407 +0100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'System': total_events='197' with empty_msg='0'. 
02-03-2014 12:34:31.028 +0100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'Application' 
02-03-2014 12:34:31.028 +0100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'Application': total_events='89' with empty_msg='0'. 
02-03-2014 12:34:31.278 +0100 INFO WinEventLogInputProcessor - main-thread: Processing existing Windows Event Log 'System' 
02-03-2014 12:34:31.278 +0100 INFO WinEventLogInputProcessor - main-thread: Finished processing existing Windows Event Log 'System': total_events='201' with empty_msg='0'.

Which is the reason? Is it because the customer uses a custom app to collect the windows event logs?

I have got some questions regarding the SDDL configuration which I would like to validate for the security logs. The customer has provided me the wevtutil output for the Windows 2012 server, but that tool doesn't exist for Windows 2003.

Customer has affirmed that now all necessary permissions have been granted, but this has not changed the situation. Customer seems to search in the proper way on the IX (he sees for example Application and System events in splunkweb, but no Security events).

On the metrics.log of the UF there are several events about Security Windows:

02-24-2014 14:23:08.401 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.427035, eps=1.161358, kb=13.237305, ev=36, avg_age=1.222222, max_age=2 
02-24-2014 14:23:08.401 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.427035, eps=1.161358, kb=13.237305, ev=36, avg_age=1.222222, max_age=2 
02-24-2014 14:27:16.387 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.042846, eps=0.129043, kb=1.328125, ev=4, avg_age=1.000000, max_age=1 
02-24-2014 14:27:16.387 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.042846, eps=0.129043, kb=1.328125, ev=4, avg_age=1.000000, max_age=1 
02-24-2014 14:57:13.183 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.042258, eps=0.128978, kb=1.310547, ev=4, avg_age=3.000000, max_age=3 
02-24-2014 14:57:13.183 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.042258, eps=0.128978, kb=1.310547, ev=4, avg_age=3.000000, max_age=3 
02-24-2014 15:01:10.586 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129043, kb=3.191406, ev=4, avg_age=3.000000, max_age=3 
02-24-2014 15:01:10.586 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129043, kb=3.191406, ev=4, avg_age=3.000000, max_age=3 
02-24-2014 15:02:12.597 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129043, kb=3.191406, ev=4, avg_age=2.000000, max_age=2 
02-24-2014 15:02:12.597 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129043, kb=3.191406, ev=4, avg_age=2.000000, max_age=2 
02-24-2014 15:03:45.590 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129042, kb=3.191406, ev=4, avg_age=2.000000, max_age=2 
02-24-2014 15:03:45.590 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.102957, eps=0.129042, kb=3.191406, ev=4, avg_age=2.000000, max_age=2 
02-24-2014 15:08:24.599 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=192.231400, eps=249.829675, kb=5961.697266, ev=7748, avg_age=0.527104, max_age=2 
02-24-2014 15:08:24.599 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=192.231400, eps=249.829675, kb=5961.697266, ev=7748, avg_age=0.527104, max_age=2 
02-24-2014 15:08:55.596 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=124.711407, eps=162.593578, kb=3865.746094, ev=5040, avg_age=0.584722, max_age=2 
02-24-2014 15:08:55.596 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=124.711407, eps=162.593578, kb=3865.746094, ev=5040, avg_age=0.584722, max_age=2 
02-24-2014 15:09:26.594 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=119.831783, eps=156.270475, kb=3714.490234, ev=4844, avg_age=0.465731, max_age=1 
02-24-2014 15:09:26.594 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=119.831783, eps=156.270475, kb=3714.490234, ev=4844, avg_age=0.465731, max_age=1 
02-24-2014 15:17:24.910 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.022808, eps=0.067405, kb=0.676758, ev=2, avg_age=78.000000, max_age=78 
02-24-2014 15:17:24.910 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.022808, eps=0.067405, kb=0.676758, ev=2, avg_age=78.000000, max_age=78 
02-24-2014 15:28:15.905 +0100 INFO Metrics - group=per_source_thruput, series="wineventlog:security", kbps=0.021612, eps=0.064521, kb=0.669922, ev=2, avg_age=2.000000, max_age=2 
02-24-2014 15:28:15.905 +0100 INFO Metrics - group=per_sourcetype_thruput, series="wineventlog:security", kbps=0.021612, eps=0.064521, kb=0.669922, ev=2, avg_age=2.000000, max_age=2

Does this necessarily mean that Security events are forwarded or is this not a reliable information?

In addition I noticed that while on the Windows 2003 UF (where Security Windows Event logs are successfully collected) no persistentstorage folder at all exists, on the Windows 2008 and Windows 20012 UFs (where Security Windows Event logs are not successfully collected) the persistentstorage folder and all checkpoint files exist, are written and constantly updated. If you stop splunk, delete che Security checkpoint file and then restart splunk, the file gets recreated and its content dynamically updated, but no Security events are seen on the IX anyway.

At this point I have several additional questions:

1. is there any tool for retrieving the information regarding the security event log in Windows 2003, especially regarding the information related to the isolation mode and the channel security descriptor?
2. is there a official way or document to make sure that the provided wevtutil output for the Windows 2012 is a valid configuration which should allow the security logs to be collected and processed by Splunk?
3. is there any documentation which explains more in detail how the security logs permissions on Windows 2003, Windows 2008, Windows 2012 etc. should be configured in order to have the security logs successfully collected and processed by Splunk?
4. Is there a way to make sure that the user read/write permissions to the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Security have been set properly so that Splunk can collect Security logs?
5. For which reason is on the working UF the persistent storage folder missing and on the not working UFs the persistent storage folder existing with the checkpoints properly created and dynamically updated?
6. What does the RecordId inside the Windows Security checkpoint file correspond to in the Windows Event Viewer? To the EventRecordID of the event in the Security Windows event log maybe? I could not find any documentation about details of checkpoint files at all. The customer is using Splunk 6.

Thanks in advance for your help.

[mgaraventa_splu · Accepted solution]

After cleaning up the whole customer configuration (there were several unnecessary/unused/improper apps), it turned out to be something much more simple: a specific filtering (using props.conf and transforms.conf) had been put in place by the customer long time ago (he totally forgot about) for some type of Security events and for some specific UFs, so that those events were filtered out. As these UFs were not generating any other type of Security events than the ones filtered out, the customer began to think that Windows Security Logs were not collected at all, while the truth is that it has always been working as expected.

Thanks all for your support.

[daniel_splunk]

There is a known issue posted for Windows 2003, you may check whether it is related.

http://docs.splunk.com/Documentation/Splunk/6.0/ReleaseNotes/KnownIssues#Windows-specific_issues

The sourcetypes of WinEventLog on Windows 2003 is in small cases, while others have a capitalized one (example : WinEventLog:Security versus WinEventLog:security). Causing filters setup in props.conf to not match, see the workaround in answers (SPL-78726)

[ahall_splunk]

In order to collect the Windows Security Log, the user that is running the Universal Forwarder needs access to the registry that deals with the Security log. By default LOCAL SYSTEM and NETWORK SERVICE have this permission - everyone else needs to be given it. Note that if you install the Universal Forwarder as LOCAL SYSTEM, then, of course, you get to read the Windows Security Event Log by default.

To ensure the proper permissions:

• Add the user to the Event Log Readers local group
• Give the user read/write permissions to the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Security

Both of these things need to be done for a process to read the Security log. Given that other event logs work, I suspect the former is done but not the latter.

[ahall_splunk]

Unfortunately, this has become a "we need more information than can be provided in an answer" situation. There is just not enough information nor access to the environment to solve this here.

[mgaraventa_splu]

Thanks for all the replies, however after several attempts and checks the issue is still present. I am adding more details to my post description, thanks.