Discrepancy in the transfer of WinEventLog://Security logs through Universal Forwarder

[mbadhusha_splun]

We have 5 indexers and a standalone search head with no cluster configuration. Recently, we have observed that the WinEventLog: Security logs are not indexing properly and there seems to be a huge delay in indexing them.

However, the other wineventlogs such as Application & System logs are indexing as expected. There is no recent change for this on the universal forwarder (UF) nor on the indexer. We tried to check one incident for which we have not received any logs (log was in the server) then this issue came to know.

We have Splunk 6.6.2 installed on both the UF & indexers. Any idea what could be causing the issue?

[mbadhusha_splun · Accepted solution]

Made the below changes on the UF and restarted. Post which the Security logs were read from the oldest logs and the UF took some time to catch up with the latest logs.

Under Program Files/SplunkUniversalforwarder/etc/system/local/inputs.conf, add

[WinEventLog://Security]
evt_ad_cache_exp = 1200
evt_ad_cache_exp_neg = 1200
evt_ad_cache_max_entries = 40000
evt_sid_cache_exp = 300
evt_sid_cache_exp_neg = 300
evt_sid_cache_max_entries = 4000
checkpointinterval=300
use_old_eventlog_api = 1
evt_dc_name = localhost

Under Program Files/SplunkUniversalforwarder/etc/system/local/outputs.conf, add

[tcpout:primary_indexers]
tcpSendBufSz = 512000

The above are the configurations suggested by the Splunk engineering team for the delay in indexing Windows security logs. The issue has been resolved after the changes.

[harsmarvania57]

Hi,

Based on http://docs.splunk.com/Documentation/Forwarder/6.6.2/Forwarder/KnownIssues , you might hitting below known issue

2015-04-14  SPL-99687, SPL-129637   Splunk universal forwarder is 7-10 days behind recent Windows Security and system log events.

Workaround:
To mitigate this, edit the following stanza in inputs.conf: [WinEventLog://Security] evt_resolve_ad_obj = 0.