Windows Security Logs not forwarding to Splunk Cloud

[WumboJumbo675]

Have UFs configured on several Domain Controllers that point to a Heavy Forwarder and that points to Splunk Cloud. Trying to configure Windows Event Logs. Application, System & DNS logs are working correctly, however, no Security logs for any of the DCs are working.

Splunk service is running with a service account that has proper admin permissions. I have edited the DC GPO to allow the service account access to 'Manage auditing and security logs'

I am at a lose here. Not sure what else to troubleshoot.

Here is in inputs.conf file on each DC

[WinEventLog://Application]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = wineventlog

[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = wineventlog

[WinEventLog://System]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = wineventlog

[WinEventLog://DNS Server]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
index = wineventlog

[PickleRick]

If everything else works OK (other logs are ingested properly), it seems to be a local permissions problem. You can try to check the _internal events from this forwarder but I don't remember if the eventlog access problems show up in the logs if you don't raise debugging levels.

[danielcj]

Hello @WumboJumbo675 ,

1 - Confirm in Splunk Cloud if the internal logs from Heavy Forwarder are being indexed (I believe yes, since you said some logs are correct). If yes, the issue is between UFs > HFs communication.

index=_internal host=<host_name_heavy_forwarder>

2 - Confirm if the communication between UFs and HFs is working correctly. Look for ERROR messages or tcpout error messages in the UFs: 

$SPLUNK_HOME/var/log/splunk/splunkd.log

 3 - Execute a btool check to confirm if there are no syntax errors on the .conf files on UFs:

splunk btool check

4 - Check the precedence of the inputs.conf files using btool to confirm that the inputs are being read:

splunk btool inputs list --debug

5 - Confirm if there is a "wineventlog" index created in Splunk Cloud.

Let me know if this helps.

Thanks.

[WumboJumbo675]

Thanks for the response!

When I search logs for the heavy forwarder, I see the below TCP error message

- WARN AutoLoadBalancedConnectionStrategy [7892 TcpOutEloop] - Current dest host connection 44.218.224.52:9997

There are no connection errors within the slunkd.logs on the DCs

Confirmed no syntax errors and the inputs lists output is correct

Confirmed there is an index for 'wineventlog' as that is where the application/system/DNS logs are flowing.

Now that I think about it, I made the permission changes for the Splunk service account to be able to access logs on the DCs, but never rebooted them. I am wondering if a reboot is required to apply the changes... To bad I cannot reboot any of the DCs until there scheduled reboot date.

[danielcj]

Hello @WumboJumbo675 ,

If there are no errors in the connection between UFs > HFs > Splunk Cloud, there are no syntax errors, the index is created and the precedente of the inputs is correct I believe that a reboot is a good option.

Thanks.