Issues with Splunk Universal Forwarder Sending Security Logs After Upgrade to 10.0

[telvinwells08]

Hi Splunk Community,

I recently upgraded my Splunk Universal Forwarders from version 9.4.3 to 10.0, and since the upgrade, I’ve been experiencing issues with the forwarders sending security logs to my Splunk Enterprise instance (which is also running version 10.0).

Here are some specific details:

• Pre-upgrade, everything was working fine, and security logs were being ingested without any issues.
• After the upgrade, I noticed that security logs are either not getting sent or are being delayed significantly.
• I've verified that the forwarders are still forwarding some logs, but the security-related ones aren't appearing in the index as expected.
• The configuration files (inputs.conf, outputs.conf, etc.) on the forwarders haven’t been changed since the upgrade.

I’ve tried restarting the forwarders and re-checking the connectivity to the Splunk Enterprise instance, but the issue persists.

Has anyone else encountered similar problems after upgrading to 10.0? Could it be an issue with compatibility, or is there something specific I should look into? Any advice or troubleshooting tips would be greatly appreciated!

Thanks in advance for your help!

[Didalready]

Adding this is case someone has the issue, I upgrader to forwarder to 9.4.4, not 10 but reading this sounds the same. The forwarder ran as system before now runs as local account SplunkForwarder, SplunkForwarder is part of everyone, Our ad audit policy rules had some with everyone on read of anything. this rule caused the security log to log 100k 4662 of splunkforwarder reading an object, that looped upon itself. This caused security events not to forward at times. Reset the forwarder fixed but only for a while. My fix was to change the AD audit policy to be for domain users not everyone.  My security events now are loaded timely. To see if this is your issues look in security logs for 4662, by account running the forwarder, if  you see thousands of events in a few seconds now and then, this was how I found my issue. Below someone posted to disable lookup which works but wasn't the solution for us because we delete and recreate computer objects all day.   

Hope this helps someone as it took me awhile to find our issue. 

[taylormccrary]

Hello,

I was having the same issues. In the WinEventLog://Security stanza, I changed evt_resolve_ad_obj from 1 to 0 and it finally started working again. Hope this helps!

evt_resolve_ad_obj = 0

[SDSplQuestion]

Hi TelvinWell08,

I've had a similar issue recently, did you end up finding out a resolution to this one? Can't seem to get them to send data again?

[livehybrid]

Hi @telvinwells08 

From my experience there shouldnt be anything that would cause this issue, however Im wondering if there is something else causing these delayed/missed logs. 

Have you been able to check $SPLUNK_HOME/var/log/splunk/splunkd.log ? Are there any errors or specific logs relating to security or sending of data which might indicate the cause of the delay? Feel free to share any errors here and we can look into the for you.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing