topic Re: Why does Splunk upgrade to version 9.1.0.1 ends prematurely? in Installation

Why does Splunk upgrade to version 9.1.0.1 end prematurely?

Gregski11 — Wed, 02 Aug 2023 13:25:53 GMT

trying to upgrade our Windows Server 2019 based Splunk version 9.0.0 to 9.1.0.1 and it's randomly failing on 50% or half of our 12 servers in our lab

the error below is from one of our non clustered Search Heads, others which are identical installed fine, we got the same error on our index Cluster Master

Splunk Enterprise Setup Wizard ended prematurely

Splunk Enterprise Setup Wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time, run Setup Wizard again. Click the Fiinish button to exit the Setup Wizard.

Setup cannot copy the following files:

Splknetdrv.sys
SplunkMonitorNoHandleDrv.sys
SplunkDrv.sys

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

PickleRick — Mon, 31 Jul 2023 19:19:08 GMT

Launch it with verbose logging and check the logfile for errors/warnings

https://docs.splunk.com/Documentation/Splunk/9.1.0/Installation/InstallonWindowsviathecommandline#Install_Splunk_Enterprise_with_verbose_logging_to_C:.5CTEMP.5CSplunkInstall.log

As a quick check - do your servers all run as Local System or maybe you have some permission issues?

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

Gregski11 — Mon, 31 Jul 2023 20:22:56 GMT

ok, did just that and here is a section from the install log where things go south and setup begins rolling back

MSI (s) (FC:F4) [12:47:17:625]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF1EF.tmp, Entrypoint: StopSplunkServiceDefCA
MSI (s) (FC:68) [12:47:17:625]: Generating random cookie.
MSI (s) (FC:68) [12:47:17:625]: Created Custom Action Server with PID 2452 (0x994).
MSI (s) (FC:70) [12:47:17:719]: Running as a service.
MSI (s) (FC:70) [12:47:17:719]: Hello, I'm your 64bit Elevated Non-remapped custom action server.
StopSplunkServiceDef: Warning: Invalid property ignored: FailCA=.
StopSplunkServiceDef: Info: Properties: splunkHome: C:\Program Files\Splunk, svcName: Splunkd.
StopSplunkServiceDef: Info: Enter.
StopSplunkServiceDef: Info: service Splunkd already exists
StopSplunkServiceDef: Info: Leave.
MSI (s) (FC:F4) [12:47:17:734]: Executing op: ActionStart(Name=ReinstallRegmonDrv,,)
Action 12:47:17: ReinstallRegmonDrv.
MSI (s) (FC:F4) [12:47:17:734]: Executing op: CustomActionSchedule(Action=ReinstallRegmonDrv,ActionType=1281,Source=BinaryData,Target=InstallRegmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:17:750]: Executing op: ActionStart(Name=UninstallNetmonDrv,,)
Action 12:47:17: UninstallNetmonDrv.
MSI (s) (FC:F4) [12:47:17:750]: Executing op: CustomActionSchedule(Action=UninstallNetmonDrv,ActionType=3073,Source=BinaryData,Target=UninstallNetmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:14) [12:47:17:812]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF2AB.tmp, Entrypoint: UninstallNetmonDrvCA
UninstallNetmonDrv: Warning: Invalid property ignored: FailCA=.
UninstallNetmonDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\splknetdrv.inf.
UninstallNetmonDrv: Info: Enter.
UninstallNetmonDrv: Info: Service: splknetdrv, state: 1.
UninstallNetmonDrv: Info: splknetdrv service does not exists.
UninstallNetmonDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf
UninstallNetmonDrv: Info: SystemPath is: C:\Windows\system32\
UninstallNetmonDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallNetmonDrv: Info: WaitForSingleObject returned : 0x0
UninstallNetmonDrv: Info: Exit code for process : 0x0
UninstallNetmonDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:18:390]: Executing op: ActionStart(Name=ReinstallNohandleDrv,,)
Action 12:47:18: ReinstallNohandleDrv.
MSI (s) (FC:F4) [12:47:18:390]: Executing op: CustomActionSchedule(Action=ReinstallNohandleDrv,ActionType=1281,Source=BinaryData,Target=InstallNohandleDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:18:390]: Executing op: ActionStart(Name=UninstallRegmonDrv,,)
Action 12:47:18: UninstallRegmonDrv.
MSI (s) (FC:F4) [12:47:18:406]: Executing op: CustomActionSchedule(Action=UninstallRegmonDrv,ActionType=3073,Source=BinaryData,Target=UninstallRegmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:50) [12:47:18:453]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF53D.tmp, Entrypoint: UninstallRegmonDrvCA
UninstallRegmonDrv: Warning: Invalid property ignored: FailCA=.
UninstallRegmonDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\splunkdrv.inf.
UninstallRegmonDrv: Info: Enter.
UninstallRegmonDrv: Info: Service: splunkdrv, state: 1.
UninstallRegmonDrv: Info: splunkdrv service does not exists.
UninstallRegmonDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf
UninstallRegmonDrv: Info: SystemPath is: C:\Windows\system32\
UninstallRegmonDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallRegmonDrv: Info: WaitForSingleObject returned : 0x0
UninstallRegmonDrv: Info: Exit code for process : 0x0
UninstallRegmonDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:19:031]: Executing op: ActionStart(Name=ReinstallNetmonDrv,,)
Action 12:47:19: ReinstallNetmonDrv.
MSI (s) (FC:F4) [12:47:19:047]: Executing op: CustomActionSchedule(Action=ReinstallNetmonDrv,ActionType=1281,Source=BinaryData,Target=InstallNetmonDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:F4) [12:47:19:047]: Executing op: ActionStart(Name=UninstallNohandleDrv,,)
Action 12:47:19: UninstallNohandleDrv.
MSI (s) (FC:F4) [12:47:19:047]: Executing op: CustomActionSchedule(Action=UninstallNohandleDrv,ActionType=3073,Source=BinaryData,Target=UninstallNohandleDrvCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;FailCA=)
MSI (s) (FC:88) [12:47:19:094]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIF7BE.tmp, Entrypoint: UninstallNohandleDrvCA
UninstallNohandleDrv: Warning: Invalid property ignored: FailCA=.
UninstallNohandleDrv: Info: Driver inf file: C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf.
UninstallNohandleDrv: Info: Enter.
UninstallNohandleDrv: Info: Service: SplunkMonitorNoHandle, state: 1.
UninstallNohandleDrv: Info: SplunkMonitorNoHandle service does not exists.
UninstallNohandleDrv: Info: Enter. Args: rundll32.exe, setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf
UninstallNohandleDrv: Info: SystemPath is: C:\Windows\system32\
UninstallNohandleDrv: Info: Execute string: C:\Windows\system32\cmd.exe /c "C:\Windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\MUSZYN~1\AppData\Local\Temp\splunk.log" 2>&1"
UninstallNohandleDrv: Info: WaitForSingleObject returned : 0x0
UninstallNohandleDrv: Info: Exit code for process : 0x0
UninstallNohandleDrv: Info: Leave.
MSI (s) (FC:F4) [12:47:19:703]: Executing op: ActionStart(Name=RemoveFiles,Description=Removing files,Template=File: [1], Directory: [9])
Action 12:47:19: RemoveFiles. Removing files
MSI (s) (FC:F4) [12:47:19:703]: Executing op: ProgressTotal(Total=17609,Type=1,ByteEquivalent=175000)
MSI (s) (FC:F4) [12:47:19:703]: Executing op: SetTargetFolder(Folder=C:\ProgramData\Splunk Enterprise\)

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

jho-splunk — Tue, 01 Aug 2023 14:45:43 GMT

HI @Gregski11 ,

Everything appears be fine in the snippet you posted. Please see here for instructions on how to troubleshoot further: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#M7187.

Cheers,

- Jo.

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

Gregski11 — Wed, 02 Aug 2023 00:47:24 GMT

Rollback: StopSplunkServiceDef
MSI (s) (B4:98) [22:00:59:416]: Executing op: ActionStart(Name=StopSplunkServiceDef,,)
Rollback: RestartSplunkService
MSI (s) (B4:98) [22:00:59:416]: Executing op: ActionStart(Name=RestartSplunkService,,)
MSI (s) (B4:98) [22:00:59:416]: Executing op: CustomActionRollback(Action=RestartSplunkService,ActionType=1281,Source=BinaryData,Target=StartSplunkServiceCA,CustomActionData=SystemFolder=C:\Windows\SysWOW64\;System64Folder=C:\Windows\system32\;SplunkHome=C:\Program Files\Splunk\;SplunkSvcName=Splunkd;LaunchSplunk=1;FailCA=)
MSI (s) (B4:BC) [22:00:59:478]: Invoking remote custom action. DLL: C:\Windows\Installer\MSIDCBF.tmp, Entrypoint: StartSplunkServiceCA
StartSplunkService: Warning: Invalid property ignored: FailCA=.
StartSplunkService: Info: Properties: splunkHome: C:\Program Files\Splunk, svcName: Splunkd, launch splunk: 1.
StartSplunkService: Info: Enter.
StartSplunkService: Info: service Splunkd already exists
StartSplunkService: Info: Leave.
StartSplunkService: Info: Enter. Args: "C:\Program Files\Splunk\bin\splunk.exe", start --answer-yes --no-prompt --accept-license --auto-ports
StartSplunkService: Info: SystemPath is: C:\Windows\system32\
StartSplunkService: Info: Execute string: C:\Windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
StartSplunkService: Info: WaitForSingleObject returned : 0x0
StartSplunkService: Info: Exit code for process : 0x1
StartSplunkService: Info: Leave.
StartSplunkService: Error: ExecCmd failed: 0x1.
StartSplunkService: Error 0x80004005: Cannot start splunkd service.
CustomAction RestartSplunkService returned actual error code 1603 but will be translated to success due to continue marking
Rollback: Updating component registration

Re: Why does Splunk upgrade to version 9.1.0.1 ends prematurely?

Gregski11 — Wed, 02 Aug 2023 01:12:49 GMT

looks like people were having the same problem six years ago

https://community.splunk.com/t5/Building-for-the-Splunk-Platform/Rollback-during-Installation-Windows-64-bit/m-p/335020/highlight/false#M4994

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

jho-splunk — Wed, 02 Aug 2023 09:07:02 GMT

Hi @Gregski11,

Hmmm...not much to go on there unfortunately. Is there a %TEMP%\splunk.log? How long does it take to fail to start? There may be salient errors and/or warnings in splunkd.log and/or web_service.log. If there's nothing there, then we'll probably need to resort to a Process Monitor trace, but unfortunately that's not typically very shareable here.

Are you on the splunk-usergroups Slack, perchance?

Cheers,

- Jo.

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

_alex — Fri, 11 Aug 2023 09:20:23 GMT

having the same errors when upgrading to 9.1.0.2

have you solved it meanwhile ?

thanks

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

jho-splunk — Sat, 12 Aug 2023 23:31:45 GMT

Hi @_alex,

Unfortunately installation failures typically end very generically so it's impossible to know what happened without more information.

Please see here for instructions on how to troubleshoot further: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#....

Cheers,

- Jo.

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

lowcrawl — Tue, 22 Aug 2023 15:01:39 GMT

I was having this issue going from 9.0.4.1 to 9.1.0.1 and had to go to PowerShell and run msiexec.exe /f splunk-9.0.4.1 this repaired whatever was wrong with the current install to allow me to run msiexec.exe /i splunk-9.1.0.1

Hope this helps.

Re: Splunk upgrade to version 9.1.0.1 ends prematurely

JuriJ — Mon, 28 Aug 2023 11:13:27 GMT

Hi,

For which user does the installer and service work?

It looks like the user does not have file permissions.

The installer attempts to run the SPLUNK process after installation. If the Splunk process does not start running, the installer makes the assumption that the installation failed then the installer rolls back the installation and removes the Splunk Enterprise instance.

If you use domain user or MSA then this account does not have NTFS permissions for Splunk Enterprise installation directory. After installation, you need explicitly assign NTFS permissions from that directory and all subdirectories for the MSA account.

However, you cannot do this during installation if you run the msi file directly, and as a result you will get the error that is mentioned above.

Solution:

Install Splunk from the command line and use the LAUNCHSPLUNK=0 flag to keep Splunk Enterprise from starting after installation has completed.

For example :

PS C:\temp> msiexec.exe /i splunk-9.0.4-de405f4a7979-x64-release.msi LAUNCHSPLUNK=0

You can complete the installation, and before running SPLUNK, you need to grant the user "Full Control" permissions to the Splunk Enterprise installation directory and all of its subdirectories.

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

Gregski11 — Tue, 17 Oct 2023 14:20:40 GMT

I don't like abandoned threads, so after hours and hours with Splunk Tech Support and having opened multiple cases on this issue I have something worth sharing.

You may need to do Two things to get this to work / install:

First Part

1. using the command line stop Splunk from running, so run SPLUNK STOP on Windows it may look somethign like this

C:\Program Files\Splunk\bin>splunk stop

2. then run the install MSI with the LAUNCHSPLUNK=0 parameter (no it does not have to be all in caps, not on Windows anyways)

C:\Software\Splunk>splunk-9.1.1-64e843ea36b1-x64-release.msi launchsplunk=0

Alright, that may get it installed, but afterwards you will notice it does not want to start, but that's ok, I will show you what to do to get it to start in the follow up post.

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

MKhan1 — Thu, 18 Jul 2024 19:07:07 GMT

Hi Gregski,

I was able to get Splunk to install via your steps, but it wont start as you mentioned. What steps do I need to do to get the SplunkD service to start. Currently i get a timeout error when attempting to start it via CLI

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

sideview — Fri, 18 Apr 2025 20:57:10 GMT

We saw this problem with a customer deployment as well. It turned out that a different admin, not the main admin who was usually on the box, had set Splunkd some time ago to only be run as a certain domain user rather than as system.

The msi's upgrade at the end restarts splunk but I guess it ends up restarting as the user who ran the msi, so it fails. Another clue was that restarting splunkd on the command line, by the administrator user, failed with "splunk stopped" as the only output.

tacking launchsplunk=0 onto the msi invocation was the answer ultimately.

and then the admins set Splunk back to just run as System so it wouldn't cause any unexpected problems going forward

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

BelCySecConsult — Thu, 31 Jul 2025 17:41:43 GMT

I don't see where you address your the issue of the service not starting after you complete the install "Alright, that may get it installed, but afterwards you will notice it does not want to start, but that's ok, I will show you what to do to get it to start in the follow up post."

Re: Why does Splunk upgrade to version 9.1.0.1 end prematurely?

jho-splunk — Fri, 01 Aug 2025 09:56:58 GMT

Hi @MKhan1,

Using LAUNCHSPLUNK=0 just ignores any problem during Splunk starting up, so unfortunately it still could be pretty much anything. If you run splunk start from a command line, what does it say? Have you tried checking EventLogs and splunkd.log?

Cheers,

- Jo.