Update error from 9.0.4 to 9.2.0.1

[L_Petch]

Hello All,

I am currently testing upgrading from Splunk Enterprise version 9.0.4 to 9.2.0.1 but get the below error. 

Traceback (most recent call last):

File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py",  line 39, in <module>

from splunk.rcUtils import makeRestCall, CliArgError, NoEndPointError, InvalidStatusCodeError

MemoryError

Error running pre-start tasks. 

I will add that there are a few more lines to the error but this is an air-gapped environment and hoping there is no need to manually type it all out 🙂

TIA

Leon

[steveski]

Had this same error, after uninstalling the Splunk forwarder, then installing Splunk enterprise on RHEL 9 linux.  Rebooted the system then ran the install again and had no reported memory errors.   The above python script fix will work, but a reboot could work as well. 

[steveski]

Update, Memory error occurred again when trying to restart Splunk, updated the .py script.  My system has 12 GB of Free RAM.   I've seen this error now on 2 different Splunk 9.2.1 installs.

[Darthsplunker]

Ran into exact same issue going from 9.1.1 to 9.2.1. had the same errors and commented out the line "marnall" pointed out, restarted splunk ,finished the update and successfully started Splunk..

Why is this happening? and does it happen to everyone?

[marnall]

The word MemoryError is concerning. Usually that means Python thinks it will run out of memory. Do you have a comfortable amount of RAM on the machine?

[L_Petch]

Hi Marnell,

Yes this error is happening on my 2 HF's and deployment servers. All have 12GB of RAM with 10GB available.

[marnall]

Are you previewing the upgrade or doing it for real? If so, could you outline the commands that you used, and summarize the results returned by the machine?

[L_Petch]

I have tried both the preview and upgrading without doing the preview. It is after running the start command that it errors.

In preview mode I get the same error it just takes a lot longer to get to the error.

- rpm -U splunk_package_name.rpm

- $SPLUNK_HOME/bin/splunk start

[marnall]

Weird. Could you check if there are any other errors? If it's just the MemoryError then you could try measuring the memory usage at the time of the upgrade in case anything explodes in memory usage during the upgrade.

[L_Petch]

This is the full error output,

Migrating to:
VERSION=9.2.0.1
BUILD=d8ae995bf219
PRODUCT=splunk
PLATFORM=Linux-x86_64

********** BEGIN PREVIEW OF CONFIGURATION FILE MIGRATION **********

Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py", line 39, in <module>
from splunk.rcUtils import makeRestCall, CliArgError, NoEndpointError, InvalidStatusCodeError
File "/opt/splunk/lib/python3.7/site-packages/splunk/rcUtils.py", line 17, in <module>
from splunk.search import dispatch, getJob, listJobs
File "/opt/splunk/lib/python3.7/site-packages/splunk/search/__init__.py", line 17, in <module>
from splunk.rest.splunk_web_requests import is_v2_search_enabled
File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/splunk_web_requests/__init__.py", line 1, in <module>
import cherrypy
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/__init__.py", line 76, in <module>
from . import _cprequest, _cpserver, _cptree, _cplogging, _cpconfig
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpserver.py", line 6, in <module>
from cherrypy.process.servers import ServerAdapter
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/__init__.py", line 13, in <module>
from .wspbus import bus
File "/opt/splunk/lib/python3.7/site-packages/cherrypy/process/wspbus.py", line 66, in <module>
import ctypes
File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 551, in <module>
_reset_cache()
File "/opt/splunk/lib/python3.7/ctypes/__init__.py", line 273, in _reset_cache
CFUNCTYPE(c_int)(lambda: None)
MemoryError
Error running pre-start tasks

[marnall · Accepted solution]

This is an odd error. Could you try opening the file at "/opt/splunk/lib/python3.7/ctypes/__init__.py", then commenting the line:

CFUNCTYPE(c_int)(lambda: None)

It should be on line 273. Put a hash before it, so your _reset_cache() function looks like this:

def _reset_cache():
    _pointer_type_cache.clear()
    _c_functype_cache.clear()
    if _os.name == "nt":
        _win_functype_cache.clear()
    # _SimpleCData.c_wchar_p_from_param
    POINTER(c_wchar).from_param = c_wchar_p.from_param
    # _SimpleCData.c_char_p_from_param
    POINTER(c_char).from_param = c_char_p.from_param
    _pointer_type_cache[None] = c_void_p
    # XXX for whatever reasons, creating the first instance of a callback
    # function is needed for the unittests on Win64 to succeed.  This MAY
    # be a compiler bug, since the problem occurs only when _ctypes is
    # compiled with the MS SDK compiler.  Or an uninitialized variable?
    #CFUNCTYPE(c_int)(lambda: None)

Then try testing the upgrade

[MightyJ]

This issue first occurred intermittently after upgrading from 9.0.5 to 9.2.1 Splunk Enterprise on a Linux kernel.

In place upgrade from 9.2.1 to 9.2.2 didn't fixed the issue neither

But commenting out this line as such fixed it.

#CFUNCTYPE(c_int)(lambda: None)

Also tested on another box with the same issue, I've commented out the line first and then upgraded from 9.2.1 to 9.2.2, the upgrade override the fix, and I have to re-apply the fix again.

Definitely raising a Splunk support case for this one 🙂

Thank you!

[edoardo_vicendo]

Thank you! Same issue here on Splunk 9.2.1

Splunk was NOT starting at boot-start (with init.d) but manually was starting correctly.

After having commented the mentioned line is now properly booting with the VM (Oracle Linux).

I am going to open a case to the support to inform them about it.

[edoardo_vicendo]

I had a reply from the Splunk Support, it seems that since a while init.d is not supported anymore as mentioned here:

https://docs.splunk.com/Documentation/Splunk/9.2.1/Admin/ConfigureSplunktostartatboottime

"The init.d boot-start script is not compatible with RHEL 8 and higher. You can instead configure systemd to manage boot start and run splunkd as a service. For more information, see Enable boot start on machines that run systemd."

In fact I have this issue on a Oracle Linux 8 machine.

[L_Petch]

This was indeed the issue. Thank you so much for the help.

[L_Petch]

Just to add to this I doubled the RAM in my dev environment for the VM to 24GB and still get the same error