Splunk will not start - ImportError: cannot import name simpleRequest

[rphillips_splk]

Splunk fails to start. From the cli startup we can see the following:

D:\Program Files\Splunk\bin>.\splunk status
Splunkd: Stopped

D:\Program Files\Splunk\bin>.\splunk start

Splunk> Another one.

Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking appserver port [127.0.0.1:8065]: open
        Checking kvstore port [8191]: open
Traceback (most recent call last):
  File "D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\clilib\cli.p
y", line 31, in <module>
    from splunk.rcUtils import makeRestCall, CliArgError, NoEndpointError, Inval
idStatusCodeError
  File "D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rcUtils.py",
 line 17, in <module>
    from splunk.rest import simpleRequest
ImportError: cannot import name simpleRequest

D:\Program Files\Splunk\bin>

How do we resolve this issue and get Splunk running again ?

Splunk 6.5.2 (build 67571ef4b87d)
OS Name: Microsoft Windows Server 2012 R2 Standard

[rphillips_splk · Accepted solution]

1) We can see from splunkd.log that Splunk cannot import the simpleRequest object from the splunk.rest module

splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - Traceback (most recent call last): 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - File "D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\clilib\cli.py", line 31, in <module> 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - from splunk.rcUtils import makeRestCall, CliArgError, NoEndpointError, InvalidStatusCodeError 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - File "D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rcUtils.py", line 17, in <module> 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - from splunk.rest import simpleRequest 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - ImportError: cannot import name simpleRequest 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - Bypassing local license checks since this instance is configured with a remote license master. 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - 
splunkd.log:05-24-2017 16:21:51.535 -0400 ERROR loader - <<<<< EOF (pre-flight-checks) 

2) trying to manually import the simpleRequest object fails as well

D:\Program Files\Splunk>bin\splunk cmd python 
Python 2.7.11 (default, Jan 5 2017, 00:56:47) [MSC v.1900 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information. 
>>> import os 
>>> import splunk.rest 
>>> print os.path.abspath(splunk.rest.__file__) 
D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest\__init__.pyc
>>> from splunk.rest import simpleRequest 
Traceback (most recent call last): 
File "<stdin>", line 1, in <module> 
ImportError: cannot import name simpleRequest 
>>> exit 
Use exit() or Ctrl-Z plus Return to exit 
>>> ^Z 

3) print os.path.abspath(splunk.rest.__file__)
shows that the file from which "splunk.rest" comes from is that __init__.pyc

4) Looking in the directory we see that the compiled python file (__init__.pyc) is only 163 bytes which indicates its either corrupted or truncated.

D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest>dir 
Volume in drive D is Data 
Volume Serial Number is 400C-EB2A 

Directory of D:\Program Files\Splunk\Python-2.7\Lib\site-packages\splunk\rest 

05/24/2017 03:25 PM <DIR> . 
05/24/2017 03:25 PM <DIR> .. 
04/02/2017 02:01 AM <DIR> external 
01/23/2017 12:04 AM 29,633 format.py 
04/01/2017 11:24 PM 26,033 format.pyc 
04/01/2017 11:26 PM 26,033 format.pyo 
01/23/2017 12:04 AM 496 payload.py 
04/01/2017 11:26 PM 802 payload.pyo 
01/23/2017 12:04 AM 5,155 test.py 
01/23/2017 12:04 AM 32,131 __init__.py 
**05/24/2017 03:25 PM 163 __init__.pyc** 
04/01/2017 11:26 PM 24,407 __init__.pyo 
9 File(s) 144,853 bytes 
3 Dir(s) 81,578,614,784 bytes free 

5) removing the __init__.pyc file will cause it to be recreated on the next import

D:\Program Files\Splunk>del Python-2.7\Lib\site-packages\splunk\rest\__init__.pyc 

D:\Program Files\Splunk>bin\splunk cmd python 
Python 2.7.11 (default, Jan 5 2017, 00:56:47) [MSC v.1900 64 bit (AMD64)] on win32 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from splunk.rest import simpleRequest 
>>> ^Z 

6) Once we could successfully import the object , Splunk was able to startup.

[rphillips_splk]

The root cause on this issue was determined by looking at resource_usage.log and seeing that the host was out of memory from too many searches being run. We also had several crash files at the time the .pyc file was touched at 05/24/2017 03:25 PM.
Most likely due to the memory constraints on the system the python file __init__.pyc was not able to be compiled properly therefore leaving it the truncated 163 bytes bad state.

There are several things you can do to manage the memory consumption :

• understand your concurrent search load (for this the Monitoring Console is a good tool for visibility into the deployment)
  https://docs.splunk.com/Documentation/Splunk/6.6.0/DMC/DMCoverview

• if your search demand is more than your memory can sustain, either add more memory, or move to a distributed deployment with search head clustering to spread out the search load, or reduce number of searches being run concurrently.

• avoid real time search if possible

• review the memory tracker features available in limits.conf to manage memory usage per search

https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Limitsconf

Memory tracker

This section contains settings for the memory tracker.

enable_memory_tracker =

* Specifies if the memory tracker is enabled.
* When set to “false” (disabled): The search is not terminated even if
the search exceeds the memory limit.
* When set to “true”: Enables the memory tracker.
* Must be set to “true” to enable the “search_process_memory_usage_threshold”
setting or the “search_process_memory_usage_percentage_threshold” setting.
* Default: false

search_process_memory_usage_threshold =

* To use this setting, the “enable_memory_tracker” setting must be set
to “true”.
* Specifies the maximum memory, in MB, that the search process can consume
in RAM.
* Search processes that violate the threshold are terminated.
* If the value is set to 0, then search processes are allowed to grow
unbounded in terms of in memory usage.
* Default: 4000 (4GB)

search_process_memory_usage_percentage_threshold =

* To use this setting, the “enable_memory_tracker” setting must be set
to “true”.
* Specifies the percent of the total memory that the search process is
entitled to consume.
* Search processes that violate the threshold percentage are terminated.
* If the value is set to zero, then splunk search processes are allowed to
grow unbounded in terms of percentage memory usage.
* Any setting larger than 100 or less than 0 is discarded and the default
value is used.
* Default: 25%