Why is Splunk Universal Forwarder shutting down after installation for collection of Windows AD events?

[rmn]

We are trying to collect Windows AD Events and push them to splunk cloud by following the official splunk documentation: https://docs.splunk.com/Documentation/SplunkCloud/9.0.2209/Admin/WindowsGDI

As soon as we see the windows events coming into our Splunk Cloud index wineventlog in xml format, just a minute later splunkd.log shows detected app modification in the UF app and then Shutdown HTTPDispatchThread message which shutsdown everything and the events ingestion stops. 

Following the above article, I have made no additional change to anything anywhere and really not sure what is going wrong here. Below are some logs from the UF Splunkd.log file. I will be forever grateful for your help and guidance towards the right path. 

05-03-2023 09:43:39.022 +0000 INFO DeployedApplication [8844 HttpClientPollingThread_05BA710E-180D-492D-805D-227A46000E35] - Installing app=100_company_splunkcloud to='C:\Program Files\SplunkUniversalForwarder\etc\apps\100_company_splunkcloud'
05-03-2023 09:43:39.163 +0000 INFO ApplicationManager [8844 HttpClientPollingThread_05BA710E-180D-492D-805D-227A46000E35] - Detected app modification: 100_company_splunkcloud
05-03-2023 09:43:39.257 +0000 WARN DC:DeploymentClient [8844 HttpClientPollingThread_05BA710E-180D-492D-805D-227A46000E35] - Restarting Splunkd...
05-03-2023 09:43:39.288 +0000 INFO HttpPubSubConnection [8844 HttpClientPollingThread_05BA710E-180D-492D-805D-227A46000E35] - Running phone uri=/services/broker/phonehome/connection_IP_8089_SERVERNAME_05BA710E-180D-492D-805D-227A46000E35
05-03-2023 09:43:41.731 +0000 INFO loader [4160 HTTPDispatch] - Shutdown HTTPDispatchThread
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - Shutting down splunkd
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_Begin"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_NoahHealthReport"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_FileIntegrityChecker"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_JustBeforeKVStore"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_KVStore"
05-03-2023 09:43:41.746 +0000 INFO CollectionCacheManager [8852 CollectionCacheBookkeepingThread] - CollectionCacheBookkeepingThread finished eloop
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_DFM"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_Thruput"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_FederatedHeartBeat"
05-03-2023 09:43:41.746 +0000 INFO Shutdown [4776 Shutdown] - shutting down level="ShutdownLevel_TcpInput1"
05-03-2023 09:43:41.746 +0000 INFO TcpInputProc [4776 Shutdown] - Running shutdown level 1. Closing listening ports.
05-03-2023 09:43:41.746 +0000 INFO TcpInputProc [4776 Shutdown] - Done setting shutdown in progress signal.
05-03-2023 09:43:41.746 +0000 INFO TcpInputProc [2400 TcpListener] - Shutting down listening ports

[jho-splunk]

Hi @rmn,

So are you saying that when this happens, Splunk does not automatically restart?

Cheers,

 - Jo.

[rmn]

Hi @jho-splunk ,

It does automatically restarts because I enabled the restart splunkd.restart while configurin the server classes. The processes are shutting down in the UF Client as shown in the splunkd paste above. Not sure why is that happening. 

[jho-splunk]

Hi @rmn,

Well, it should restart in these situations based on the logging...so I'm not sure I follow now.  Are you saying that it then continually restarts like every minute or something like that?

Cheers,

 - Jo.

[rmn]

Hi @jho-splunk,

It doesnt restarts when it shutdowns all the processes. 

[jho-splunk]

Hi @rmn,

Ah ok!  Well, based on that description, it definitely sounds like something is going wrong.  I would say that your best bet is to contact Splunk Support.

Cheers,

 - Jo.