Splunk upgrade fails from 7.3.2 to 8.1.2 and rolls back in windows 2016 server

[vengisa]

Hi,

I am trying to upgrade Splunk enterprise version 7.3.2 which has forwarder enabled to the new version 8.1.2

The upgrade fails and rolls back. 

I did try to run the installation as admin user. 

Can you please let me know the possible causes and how to fix this. 

I can see the following error in migration logs 

Failed cli cmd _py_internal

Any help is appreciated

[lpino]

Hi @vengisa ,

did you manage to overcome this issue?
I've run into the same problem: Splunk upgrade from 7.3.6 to 8.0.9 on Windows Server 2016, installation seems to work fine until the progress bar stops and the Installer GUI disappears (after several minutes).
This happened to me only on 1 instance (Search Head with Splunk ES installed) in PROD environment. The same step worked fine in TEST env on the "twin" server.

I also had to restore a previous snapshot since some features of Splunk didn't work anymore after the rollback to 7.3.6.

Hope you have got good news!

[jho-splunk]

Hi @lpino,

Please see the following reply for instructions on how to troubleshoot: https://community.splunk.com/t5/Installation/Install-issue-on-Server-2016/m-p/540173/highlight/true#...

Cheers,

 - Jo.

[vengisa]

End of the file has the below info:

-- Migration information is being logged to 'C:\Program Files\Splunk\var\log\splunk\migration.log.2021-02-12.15-37-56' --
Copying 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml' to 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Handling deprecated files...

Checking script configuration...

Handling Windows scripted inputs...

C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\clilib\cli.py:1066: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logger.warn("Failed cli cmd %s" % command)
Failed cli cmd _py_internal

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=8.1.2
BUILD=545206cc9f70
PRODUCT=splunk
PLATFORM=Windows-AMD64

3:38:02 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:38:03 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:38:04 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
3:39:05 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
'"C:\Program Files\Splunk\bin\splunk.exe"' is not recognized as an internal or external command,
operable program or batch file.

[jho-splunk]

Hi @vengisa,

There may be some further detail in %TEMP%\splunk.log.  Could you let us know what it says, please?

Cheers,

 - Jo.

[vengisa]

Additional information got from the log file:

InstallFiles: File: Copying new files, Directory: , Size:
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: Patch
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: Patch 4: SELECT `Patch`.`File_`, `Patch`.`Header`, `Patch`.`Attributes`, `Patch`.`Sequence`, `Patch`.`StreamRef_` FROM `Patch` WHERE `Patch`.`File_` = ? AND `Patch`.`#_MsiActive`=? ORDER BY `Patch`.`Sequence`
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: Error
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1302
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: MsiSFCBypass
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: MsiSFCBypass 4: SELECT `File_` FROM `MsiSFCBypass` WHERE `File_` = ?
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2205 2: 3: MsiPatchHeaders
MSI (s) (08:88) [15:34:46:010]: Note: 1: 2228 2: 3: MsiPatchHeaders 4: SELECT `Header` FROM `MsiPatchHeaders` WHERE `StreamRef` = ?
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: PatchPackage
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: MsiPatchHeaders
MSI (s) (08:88) [15:34:46:214]: Note: 1: 2205 2: 3: PatchPackage
Action ended 15:35:48: InstallFiles. Return value 1.
MSI (s) (08:88) [15:35:48:740]: Doing action: RollbackRegmonDrvData
MSI (s) (08:88) [15:35:48:740]: Note: 1: 2205 2: 3: ActionText
Action 15:35:48: RollbackRegmonDrvData.

[vengisa]

Thanks @jho-splunk  for the reply!

i see this at the end.. 

MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 1708
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2205 2: 3: Error
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1708
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2205 2: 3: Error
MSI (c) (E0:D0) [15:41:36:502]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (c) (E0:D0) [15:41:36:502]: Product: Splunk Enterprise -- Installation failed.

[jho-splunk]

Hi @vengisa ,

Ah, that appears to be from the msiexec.exe log file.  What we want is the splunk.log file in the temp directory.  You should be able to find it by entering %TEMP% into the Explorer address bar.  Hopefully it will have more information on the _py_internal command.

Cheers,

 - Jo.

[vengisa]

Thanks.  I see the below in the log.  Will this help.. 

-- Migration information is being logged to 'C:\Program Files\Splunk\var\log\splunk\migration.log.2021-02-11.18-00-10' --
Copying 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml' to 'C:\Program Files\Splunk\etc\myinstall\splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Checking for possible timezone configuration errors...

Handling deprecated files...

Checking script configuration...

Handling Windows scripted inputs...

C:\Program Files\Splunk\Python-3.7\Lib\site-packages\splunk\clilib\cli.py:1066: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logger.warn("Failed cli cmd %s" % command)
Failed cli cmd _py_internal

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.

Perform migration and upgrade without previewing configuration changes? [y/n] y

Migrating to:
VERSION=8.1.2
BUILD=545206cc9f70
PRODUCT=splunk
PLATFORM=Windows-AMD64

6:00:22 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:00:24 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:00:27 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultUninstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:02:15 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" start --answer-yes --no-prompt --accept-license --auto-ports >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
The system cannot find the path specified.
6:13:42 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:13:44 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:13:45 PM
cmd.exe /c "rundll32.exe setupapi,InstallHinfSection DefaultUninstall 132 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:33 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\splunkdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:36 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\splknetdrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:37 PM
C:\windows\system32\cmd.exe /c "C:\windows\system32\rundll32.exe setupapi,InstallHinfSection DefaultInstall 128 C:\Program Files\Splunk\bin\SplunkMonitorNoHandleDrv.inf >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"
6:22:39 PM
C:\windows\system32\cmd.exe /c ""C:\Program Files\Splunk\bin\splunk.exe" _internal first-time-run --answer-yes --no-prompt >> "C:\Users\ADMINI~1\AppData\Local\Temp\splunk.log" 2>&1"

[jho-splunk]

Hi @vengisa,

Drat!  I was hoping it would provide more context, but clearly it does not.  We're going to need a Procmon log to troubleshoot further, which will be very large in size, unfortunately.  Are you able to open a Support case with Splunk?

Cheers,

 - Jo.

[vengisa]

Can you please let me know how to open one.. will do that!

[Funderburg78]

Did you find what issue caused this or get it resolved?  I have a 14 server system at 2 sites running a multi-site cluster and all my servers upgraded successfully except one of my Search heads.

[lpino]

Hi, in my case there was an application not compatible with Python 3 which I forgot to update (you can check the apps to update with Python Readiness App).
Once updated the application, the Splunk upgrade worked as expected.

Hope this may help.

[jho-splunk]

Hi @vengisa,

You will need to have access to a Support Program: https://www.splunk.com/en_us/support-and-services/support-programs.html.

If you do not, we may still be able to help you, but the problem is going to be getting access to the Procmon log I'm afraid.

Cheers,

 - Jo.