Why does Splunk upgrade to version 9.1.0.1 end prematurely?

[Gregski11]

trying to upgrade our Windows Server 2019 based Splunk version 9.0.0 to 9.1.0.1 and it's randomly failing on 50% or half of our 12 servers in our lab

the error below is from one of our non clustered Search Heads, others which are identical installed fine, we got the same error on our index Cluster Master 

Splunk Enterprise Setup Wizard ended prematurely

Splunk Enterprise Setup Wizard ended prematurely because of an error.  Your system has not been modified.  To install this program at a later time, run Setup Wizard again.  Click the Fiinish button to exit the Setup Wizard.

Setup cannot copy the following files: 

Splknetdrv.sys
SplunkMonitorNoHandleDrv.sys
SplunkDrv.sys

[Gregski11]

I don't like abandoned threads, so after hours and hours with Splunk Tech Support and having opened multiple cases on this issue I have something worth sharing.

You may need to do Two things to get this to work / install:

First Part

1.  using the command line stop Splunk from running, so run SPLUNK STOP on Windows it may look somethign like this

C:\Program Files\Splunk\bin>splunk stop

2. then run the install MSI with the LAUNCHSPLUNK=0 parameter (no it does not have to be all in caps, not on Windows anyways)

C:\Software\Splunk>splunk-9.1.1-64e843ea36b1-x64-release.msi launchsplunk=0

Alright, that may get it installed, but afterwards you will notice it does not want to start, but that's ok, I will show you what to do to get it to start in the follow up post.

[MKhan1]

Hi Gregski,

I was able to get Splunk to install via your steps, but it wont start as you mentioned. What steps do I need to do to get the SplunkD service to start. Currently i get a timeout error when attempting to start it via CLI

[lowcrawl]

I was having this issue going from 9.0.4.1 to 9.1.0.1 and had to go to PowerShell and run msiexec.exe /f  splunk-9.0.4.1 this repaired whatever was wrong with the current install to allow me to run msiexec.exe /i splunk-9.1.0.1   

Hope this helps.

[_alex]

+1

having the same errors when upgrading to 9.1.0.2

have you solved it meanwhile ?

thanks

[jho-splunk]

Hi @_alex,

Unfortunately installation failures typically end very generically so it's impossible to know what happened without more information.

Please see here for instructions on how to troubleshoot further: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#....

Cheers,

  - Jo.

[Gregski11]

looks like people were having the same problem six years ago

https://community.splunk.com/t5/Building-for-the-Splunk-Platform/Rollback-during-Installation-Window...

[PickleRick]

Launch it with verbose logging and check the logfile for errors/warnings

https://docs.splunk.com/Documentation/Splunk/9.1.0/Installation/InstallonWindowsviathecommandline#In...

As a quick check - do your servers all run as Local System or maybe you have some permission issues?

[Gregski11]

ok, did just that and here is a section from the install log where things go south and setup begins rolling back


MSI (s) (FC:F4) [12:47:17:625]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF1EF.tmp, Entrypoint: StopSplunkServiceDefCA
MSI (s) (FC:68) [12:47:17:625]: Generating random cookie.
MSI (s) (FC:68) [12:47:17:625]: Created Custom Action Server with PID 2452 (0x994).
MSI (s) (FC:70) [12:47:17:719]: Running as a service.
MSI (s) (FC:70) [12:47:17:719]: Hello, I'm your 64bit Elevated Non-remapped custom action server.
StopSplunkServiceDef: Warning: Invalid property ignored: FailCA=.
StopSplunkServiceDef: Info: Properties: splunkHome: C:\Program Files\Splunk, svcName: Splunkd.
StopSplunkServiceDef: Info: Enter.
StopSplunkServiceDef: Info: service Splunkd already exists
StopSplunkServiceDef: Info: Leave.
MSI (s) (FC:F4) [12:47:17:734]: Executing op: ActionStart(Name=ReinstallRegmonDrv,,)
Action 12:47:17: ReinstallRegmonDrv.
MSI (s) (FC:F4) [12:47:17:734]: Executing op: CustomActionSchedule(Action=ReinstallRegmonDrv,ActionType=1281,Source=BinaryData,Target=InstallRegmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:17:750]: Executing op: ActionStart(Name=UninstallNetmonDrv,,)
Action 12:47:17: UninstallNetmonDrv.
MSI (s) (FC:F4) [12:47:17:750]: Executing op: CustomActionSchedule(Action=UninstallNetmonDrv,ActionType=3073,Source=BinaryData,Target=UninstallNetmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:14) [12:47:17:812]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF2AB.tmp, Entrypoint: UninstallNetmonDrvCA
UninstallNetmonDrv: Warning: Invalid property ignored: FailCA=.
UninstallNetmonDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\splknetdrv.inf.
UninstallNetmonDrv: Info: Enter.
UninstallNetmonDrv: Info: Service: splknetdrv, state: 1.
UninstallNetmonDrv: Info: splknetdrv service does not exists.
UninstallNetmonDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf
UninstallNetmonDrv: Info: SystemPath is: C:\Windows\system32\
UninstallNetmonDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallNetmonDrv: Info: WaitForSingleObject returned : 0x0
UninstallNetmonDrv: Info: Exit code for process : 0x0
UninstallNetmonDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:18:390]: Executing op: ActionStart(Name=ReinstallNohandleDrv,,)
Action 12:47:18: ReinstallNohandleDrv.
MSI (s) (FC:F4) [12:47:18:390]: Executing op: CustomActionSchedule(Action=ReinstallNohandleDrv,ActionType=1281,Source=BinaryData,Target=InstallNohandleDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:18:390]: Executing op: ActionStart(Name=UninstallRegmonDrv,,)
Action 12:47:18: UninstallRegmonDrv.
MSI (s) (FC:F4) [12:47:18:406]: Executing op: CustomActionSchedule(Action=UninstallRegmonDrv,ActionType=3073,Source=BinaryData,Target=UninstallRegmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:50) [12:47:18:453]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF53D.tmp, Entrypoint: UninstallRegmonDrvCA
UninstallRegmonDrv: Warning: Invalid property ignored: FailCA=.
UninstallRegmonDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\splunkdrv.inf.
UninstallRegmonDrv: Info: Enter.
UninstallRegmonDrv: Info: Service: splunkdrv, state: 1.
UninstallRegmonDrv: Info: splunkdrv service does not exists.
UninstallRegmonDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf
UninstallRegmonDrv: Info: SystemPath is: C:\Windows\system32\
UninstallRegmonDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallRegmonDrv: Info: WaitForSingleObject returned : 0x0
UninstallRegmonDrv: Info: Exit code for process : 0x0
UninstallRegmonDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:19:031]: Executing op: ActionStart(Name=ReinstallNetmonDrv,,)
Action 12:47:19: ReinstallNetmonDrv.
MSI (s) (FC:F4) [12:47:19:047]: Executing op: CustomActionSchedule(Action=ReinstallNetmonDrv,ActionType=1281,Source=BinaryData,Target=InstallNetmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:19:047]: Executing op: ActionStart(Name=UninstallNohandleDrv,,)
Action 12:47:19: UninstallNohandleDrv.
MSI (s) (FC:F4) [12:47:19:047]: Executing op: CustomActionSchedule(Action=UninstallNohandleDrv,ActionType=3073,Source=BinaryData,Target=UninstallNohandleDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:88) [12:47:19:094]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF7BE.tmp, Entrypoint: UninstallNohandleDrvCA
UninstallNohandleDrv: Warning: Invalid property ignored: FailCA=.
UninstallNohandleDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf.
UninstallNohandleDrv: Info: Enter.
UninstallNohandleDrv: Info: Service: SplunkMonitorNoHandle, state: 1.
UninstallNohandleDrv: Info: SplunkMonitorNoHandle service does not exists.
UninstallNohandleDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf
UninstallNohandleDrv: Info: SystemPath is: C:\Windows\system32\
UninstallNohandleDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallNohandleDrv: Info: WaitForSingleObject returned : 0x0
UninstallNohandleDrv: Info: Exit code for process : 0x0
UninstallNohandleDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:19:703]: Executing op: ActionStart(Name=RemoveFiles,Description=Removing files,Template=File: [1], Directory: [9])
Action 12:47:19: RemoveFiles. Removing files
MSI (s) (FC:F4) [12:47:19:703]: Executing op: ProgressTotal(Total=17609,Type=1,ByteEquivalent=175000)
MSI (s) (FC:F4) [12:47:19:703]: Executing op: SetTargetFolder(Folder=C:\ProgramData\Splunk Enterprise\)

[jho-splunk]

HI @Gregski11 ,

Everything appears be fine in the snippet you posted.  Please see here for instructions on how to troubleshoot further: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#....

Cheers,

  - Jo.

[Gregski11]

Rollback: StopSplunkServiceDef
MSI (s) (B4:98) [22:00:59:416]: Executing op: ActionStart(Name=StopSplunkServiceDef,,)
Rollback: RestartSplunkService
MSI (s) (B4:98) [22:00:59:416]: Executing op: ActionStart(Name=RestartSplunkService,,)
MSI (s) (B4:98) [22:00:59:416]: Executing op: CustomActionRollback(Action=RestartSplunkService,ActionType=1281,Source=BinaryData,Target=StartSplunkServiceCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;SplunkSvcName=Splunkd;LaunchSplunk=1;FailCA=)
MSI (s) (B4:BC) [22:00:59:478]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIDCBF.tmp, Entrypoint: StartSplunkServiceCA
StartSplunkService: Warning: Invalid property ignored: FailCA=.
StartSplunkService: Info: Properties: splunkHome: C:\Program Files\Splunk, svcName: Splunkd, launch splunk: 1.
StartSplunkService: Info: Enter.
StartSplunkService: Info: service Splunkd already exists
StartSplunkService: Info: Leave.
StartSplunkService: Info: Enter. Args: "C:\Program Files\Splunk\bin\splunk.exe", start --answer-yes --no-prompt --accept-license --auto-ports
StartSplunkService: Info: SystemPath is: C:\Windows\system32\
StartSplunkService: Info: Execute string: C:\Windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
StartSplunkService: Info: WaitForSingleObject returned : 0x0
StartSplunkService: Info: Exit code for process : 0x1
StartSplunkService: Info: Leave.
StartSplunkService: Error: ExecCmd failed: 0x1.
StartSplunkService: Error 0x80004005: Cannot start splunkd service.
CustomAction RestartSplunkService returned actual error code 1603 but will be translated to success due to continue marking
Rollback: Updating component registration

[jho-splunk]

Hi @Gregski11,

Hmmm...not much to go on there unfortunately.  Is there a %TEMP%\splunk.log?  How long does it take to fail to start?  There may be salient errors and/or warnings in splunkd.log and/or web_service.log.  If there's nothing there, then we'll probably need to resort to a Process Monitor trace, but unfortunately that's not typically very shareable here.

Are you on the splunk-usergroups Slack, perchance?

Cheers,

 - Jo.

[JuriJ]

Hi,

For which user does the installer and service work?

It looks like the user does not have file permissions.

The installer attempts to run the SPLUNK process after installation. If the Splunk process does not start running, the installer makes the assumption that the installation failed then the installer rolls back the installation and removes the Splunk Enterprise instance.

If you use domain user or MSA then this account does not have NTFS permissions for Splunk Enterprise installation directory. After installation, you need explicitly assign NTFS permissions from that directory and all subdirectories for the MSA account.

However, you cannot do this during installation if you run the msi file directly, and as a result you will get the error that is mentioned above.

Solution:

Install Splunk from the command line and use the LAUNCHSPLUNK=0 flag to keep Splunk Enterprise from starting after installation has completed.

For example :

PS C:\temp> msiexec.exe /i splunk-9.0.4-de405f4a7979-x64-release.msi LAUNCHSPLUNK=0

You can complete the installation, and before running SPLUNK, you need to grant the user "Full Control" permissions to the Splunk Enterprise installation directory and all of its subdirectories.