topic Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server in Installation

Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Fri, 12 Feb 2021 18:34:28 GMT

Hi,

I am trying to upgrade Splunk enterprise version 7.3.2 which has forwarder enabled to the new version 8.1.2

The upgrade fails and rolls back.

I did try to run the installation as admin user.

Can you please let me know the possible causes and how to fix this.

I can see the following error in migration logs

Failed cli cmd _py_internal

Any help is appreciated

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

jho-splunk — Fri, 12 Feb 2021 19:03:38 GMT

Hi @vengisa,

There may be some further detail in %TEMP%\splunk.log. Could you let us know what it says, please?

Cheers,

- Jo.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Fri, 12 Feb 2021 20:45:08 GMT

Thanks @jho-splunk for the reply!

i see this at the end..

MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 1708
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2205 2: 3: Error
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1708
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2205 2: 3: Error
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (c) (E0:D0) [15:41:36:502]: Product: Splunk Enterprise -- Installation failed.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Fri, 12 Feb 2021 21:16:07 GMT

Additional information got from the log file:

InstallFiles: File: Copying new files, Directory: , Size:
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: Patch
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: Patch 4: SELECT `Patch`.`File_`, `Patch`.`Header`, `Patch`.`Attributes`, `Patch`.`Sequence`, `Patch`.`StreamRef_` FROM `Patch` WHERE `Patch`.`File_` = ? AND `Patch`.`#_MsiActive`=? ORDER BY `Patch`.`Sequence`
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: Error
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1302
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: MsiSFCBypass
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: MsiSFCBypass 4: SELECT `File_` FROM `MsiSFCBypass` WHERE `File_` = ?
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: MsiPatchHeaders
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: MsiPatchHeaders 4: SELECT `Header` FROM `MsiPatchHeaders` WHERE `StreamRef` = ?
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: PatchPackage
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: MsiPatchHeaders
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: PatchPackage
Action ended 15:35:48: InstallFiles. Return value 1.
MSI (s) (08:88) [15:35:48:740]: Doing action: RollbackRegmonDrvData
MSI (s) (08:88) [15:35:48:740]: Note: 1: 2205 2: 3: ActionText
Action 15:35:48: RollbackRegmonDrvData.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

jho-splunk — Fri, 12 Feb 2021 22:44:29 GMT

Hi @vengisa ,

Ah, that appears to be from the msiexec.exe log file. What we want is the splunk.log file in the temp directory. You should be able to find it by entering %TEMP% into the Explorer address bar. Hopefully it will have more information on the _py_internal command.

Cheers,

- Jo.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Sat, 13 Feb 2021 02:22:25 GMT

Thanks. I see the below in the log. Will this help..

-- Migration information is being logged to 'C:\Program Files\Splunk\var\log\splunk\migration.log.2021-02-11.18-00-10' --
Copying 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml' to 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Checking for possible timezone configuration errors...

Handling deprecated files...

Checking script configuration...

Handling Windows scripted inputs...

C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\clilib\cli.py:1066: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logger.warn("Failed cli cmd %s" % command)
Failed cli cmd _py_internal

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=8.1.2
BUILD=545206cc9f70
PRODUCT=splunk
PLATFORM=Windows-AMD64

6:00:22 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:00:24 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:00:27 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:02:15 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
The system cannot find the path specified.
6:13:42 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:13:44 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:13:45 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:33 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:36 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:37 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:39 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" _internal first-time-run --answer-yes --no-prompt >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Sat, 13 Feb 2021 02:25:23 GMT

End of the file has the below info:

-- Migration information is being logged to 'C:\Program Files\Splunk\var\log\splunk\migration.log.2021-02-12.15-37-56' --
Copying 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml' to 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Handling deprecated files...

Checking script configuration...

Handling Windows scripted inputs...

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=8.1.2
BUILD=545206cc9f70
PRODUCT=splunk
PLATFORM=Windows-AMD64

3:38:02 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:38:03 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:38:04 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:39:05 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
'"C:\Program Files\Splunk\bin\splunk.exe"' is not recognized as an internal or external command,
operable program or batch file.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

jho-splunk — Mon, 15 Feb 2021 10:23:45 GMT

Hi @vengisa,

Drat! I was hoping it would provide more context, but clearly it does not. We're going to need a Procmon log to troubleshoot further, which will be very large in size, unfortunately. Are you able to open a Support case with Splunk?

Cheers,

- Jo.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

vengisa — Tue, 16 Feb 2021 03:08:05 GMT

Can you please let me know how to open one.. will do that!

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

jho-splunk — Tue, 16 Feb 2021 10:55:52 GMT

Hi @vengisa,

You will need to have access to a Support Program: https://www.splunk.com/en_us/support-and-services/support-programs.html.

If you do not, we may still be able to help you, but the problem is going to be getting access to the Procmon log I'm afraid.

Cheers,

- Jo.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

lpino — Tue, 18 May 2021 13:38:00 GMT

Hi @vengisa ,

did you manage to overcome this issue?
I've run into the same problem: Splunk upgrade from 7.3.6 to 8.0.9 on Windows Server 2016, installation seems to work fine until the progress bar stops and the Installer GUI disappears (after several minutes).
This happened to me only on 1 instance (Search Head with Splunk ES installed) in PROD environment. The same step worked fine in TEST env on the "twin" server.

I also had to restore a previous snapshot since some features of Splunk didn't work anymore after the rollback to 7.3.6.

Hope you have got good news!

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

jho-splunk — Tue, 18 May 2021 14:52:26 GMT

Hi @lpino,

Please see the following reply for instructions on how to troubleshoot: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#M7187

Cheers,

- Jo.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

Funderburg78 — Tue, 26 Oct 2021 16:24:14 GMT

Did you find what issue caused this or get it resolved? I have a 14 server system at 2 sites running a multi-site cluster and all my servers upgraded successfully except one of my Search heads.

Re: Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

lpino — Fri, 05 Nov 2021 10:37:39 GMT

Hi, in my case there was an application not compatible with Python 3 which I forgot to update (you can check the apps to update with Python Readiness App).
Once updated the application, the Splunk upgrade worked as expected.

Hope this may help.