How to identify user or process responsible for stopping the Splunk UF agent

[Rastegui]

I am trying to identify the user or process responsible for stopping the Splunk UF agent. What log source do I require to be able to see this.

I have unsuccessfully tried:

• Searching in internal index - You can only see the service going down. 
  • index=_internal sourcetype=splunkd host="DC*" component=Shutdown*
• Monitoring the windows system event log for forwarder shutdown event (EventCode 7036 )
  • No visibility on who performed the action.

Looking for ideas on how this can be achieve from Splunk.

[kiran_panchavat]

@Rastegui 

To pinpoint the user or process stopping the Splunk UF, you need to look beyond Splunk’s internal logs and Windows System Events alone. 

Enable and Monitor Windows Security Event Logs

• Required Log Source: Windows Security Event Log (WinEventLog:Security)
• The Security Event Log can capture events related to service control actions if auditing is enabled.
• Specifically, Event ID 4656 (with proper auditing) or Event ID 4670 (permissions changes) might indicate when a user or process interacts with the SplunkForwarder service.
• Ensure the Splunk UF is configured to forward Windows Security Event Logs

Useful Windows Security Event Log codes to monitor for identifying the user or process responsible for stopping the Splunk UF agent:

Event ID 4688: Logs the creation of a new process. This can help identify the process responsible for stopping the Splunk UF agent.
Event ID 4648: Logs the use of explicit credentials. This can help identify the user who performed the action.
Event ID 4624: Logs successful account logons. This can help track user activity.
Event ID 4625: Logs failed account logons. This can indicate unauthorized attempts to access the system.
Event ID 1102: Logs audit log clearance. This can indicate an attempt to cover tracks.

By monitoring these event codes, you should be able to get a clearer picture of the user or process responsible for stopping the Splunk UF agent.

Please check this https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/ 

[kiran_panchavat]

@Rastegui 

• The _internal index logs Splunk’s own operational data, including shutdown events. However, it typically only records that the service stopped (e.g., "Splunkd daemon is shutting down") and not who or what triggered it. This is because Splunk UF doesn’t natively track external triggers in detailit’s a lightweight agent focused on forwarding data, not auditing administrative actions.

Event ID 7036 in the Windows System Event Log indicates that a service (like SplunkForwarder) changed state (e.g., stopped), but it doesn’t consistently log the user or process responsible for stopping it. This event is generated by the Service Control Manager (SCM) and lacks the context of the initiating action unless additional auditing is enabled.

• Event ID 4688: This event logs the creation of a new process, which can help identify the process responsible for stopping the Splunk UF agent.
• Event ID 4648: This event logs the use of explicit credentials, which can help identify the user who performed the action

Audit Logs: Enable auditing on the server to capture detailed information about user actions and process executions. This can provide more visibility into who performed the action