Update error from 9.0.4 to 9.2.0.1

[L_Petch]

Hello All,

I am currently testing upgrading from Splunk Enterprise version 9.0.4 to 9.2.0.1 but get the below error. 

Traceback (most recent call last):

File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py",  line 39, in <module>

from splunk.rcUtils import makeRestCall, CliArgError, NoEndPointError, InvalidStatusCodeError

MemoryError

Error running pre-start tasks. 

I will add that there are a few more lines to the error but this is an air-gapped environment and hoping there is no need to manually type it all out 🙂

TIA

Leon

[marnall]

The word MemoryError is concerning. Usually that means Python thinks it will run out of memory. Do you have a comfortable amount of RAM on the machine?

[L_Petch]

Hi Marnell,

Yes this error is happening on my 2 HF's and deployment servers. All have 12GB of RAM with 10GB available.

[marnall]

Are you previewing the upgrade or doing it for real? If so, could you outline the commands that you used, and summarize the results returned by the machine?

[L_Petch]

I have tried both the preview and upgrading without doing the preview. It is after running the start command that it errors.

In preview mode I get the same error it just takes a lot longer to get to the error.

- rpm -U splunk_package_name.rpm

- $SPLUNK_HOME/bin/splunk start

[marnall]

Weird. Could you check if there are any other errors? If it's just the MemoryError then you could try measuring the memory usage at the time of the upgrade in case anything explodes in memory usage during the upgrade.

[L_Petch]

This is the full error output,

Migrating to:
VERSION=9.2.0.1
BUILD=d8ae995bf219
PRODUCT=splunk
PLATFORM=Linux-x86_64

********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION **********

Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py", line 39, in <module>
from splunk.rcUtils import makeRestCall, CliArgError, NoEndpointError, InvalidStatusCodeError
File "/opt/splunk/lib/python3.7/site-packages/splunk/rcUtils.py", line 17, in <module>
from splunk.search import dispatch, getJob, listJobs
File "/opt/splunk/lib/python3.7/site-packages/splunk/search/__init__.py", line 17, in <module>
from splunk.rest.splunk_web_requests import is_v2_search_enabled
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/splunk_web_requests/__init__.py", line 1, in <module>
import cherrypy
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
from cherrypy.process.servers import ServerAdapter
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
from .wspbus import bus
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
import ctypes
File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
_reset_cache()
File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
CFUNCTYPE(c_int)(lambda: None)
MemoryError
Error running pre-start tasks

[marnall · Accepted solution]

This is an odd error. Could you try opening the file at "/opt/splunk/lib/python3.7/ctypes/__init__.py", then commenting the line:

CFUNCTYPE(c_int)(lambda: None)

It should be on line 273. Put a hash before it, so your _reset_cache() function looks like this:

def _reset_cache():
    _pointer_type_cache.clear()
    _c_functype_cache.clear()
    if _os.name == "nt":
        _win_functype_cache.clear()
    # _SimpleCData.c_wchar_p_from_param
    POINTER(c_wchar).from_param = c_wchar_p.from_param
    # _SimpleCData.c_char_p_from_param
    POINTER(c_char).from_param = c_char_p.from_param
    _pointer_type_cache[None] = c_void_p
    # XXX for whatever reasons, creating the first instance of a callback
    # function is needed for the unittests on Win64 to succeed.  This MAY
    # be a compiler bug, since the problem occurs only when _ctypes is
    # compiled with the MS SDK compiler.  Or an uninitialized variable?
    #CFUNCTYPE(c_int)(lambda: None)

Then try testing the upgrade

[edoardo_vicendo]

Thank you! Same issue here on Splunk 9.2.1

Splunk was NOT starting at boot-start (with init.d) but manually was starting correctly.

After having commented the mentioned line is now properly booting with the VM (Oracle Linux).

I am going to open a case to the support to inform them about it.

[L_Petch]

This was indeed the issue. Thank you so much for the help.

[L_Petch]

Just to add to this I doubled the RAM in my dev environment for the VM to 24GB and still get the same error