Splunk Search

Couldn't able to perform any commands in Deployment Server

anandhalagaras1
Path Finder

Hi Team,

We have recently upgraded our Deployment Master server from 7.3.1 to 8.1.2 version. The upgrade seems to be success whereas if i perform any of the commands below I am getting an error. Refer below for more information.

Commands:

splunk restart

splunk start

splunk stop

splunk reload deploy-server

 

Errors:

[splunk@servername bin]$ splunk reload deploy-server
Error processing line 1 of /opt/splunk/lib/python3.7/site-packages/zc.lockfile-2.0-py3.7-nspkg.pth:

Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "/opt/splunk/lib/python3.7/importlib/util.py", line 14, in <module>
from contextlib import contextmanager
File "/opt/splunk/lib/python3.7/contextlib.py", line 5, in <module>
from collections import deque
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 579, in <module>
main()
File "/opt/splunk/lib/python3.7/site.py", line 566, in main
known_paths = addsitepackages(known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 349, in addsitepackages
addsitedir(sitedir, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 207, in addsitedir
addpackage(sitedir, name, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 178, in addpackage
import traceback
File "/opt/splunk/lib/python3.7/traceback.py", line 3, in <module>
import collections
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

 

So kindly help to check and let me know how to resolve the issue. But our Deployment server has been upgraded successfully to 8.1.2 but couldn't able to perform any commands due to this issue.

 

Whereas if we ran the command with (./) as prefix then i can able to execute all commands:

./splunk restart

./splunk start

./splunk stop

./splunk reload deploy-server

 

So kindly help me to fix this issue.

Labels (1)
0 Karma

Vardhan
Contributor

Hi @anandhalagaras1 ,

Yes remove the Splunkforwarder package and keep only the Splunk package.

In Linux, any Splunk command should run with (./) as a prefix. Whereas in windows you can run the Splunk commands without  (./) as a prefix.

0 Karma

anandhalagaras1
Path Finder

@Vardhan ,

I have removed the Splunk forwarder package from /opt and restarted the splunk services. But still I am getting the same python error.

 

i.e. Previously before upgrade I can able to ran the below commands  from any directory (without ./) and it could fetch the results.

splunk restart

splunk start

splunk reload deploy-server

 

But now Eventhough I have removed the splunk forwarder package from /opt I couldn't able to execute the command from any directory and especially without ./

So kindly guide me how to fix it.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

What is output of: which splunk

0 Karma

anandhalagaras1
Path Finder

The output is pointing out to /opt/splunk.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Is it pointing to /opt/splunk/bin/splunk or /opt/splunk?

What you have in your $PATH?

Maybe you should create alias splunk_ds = /opt/splunk/bin/splunk and use it instead of splunk?

0 Karma

anandhalagaras1
Path Finder

@isoutamo 

If i ran the below command it is pointing to /opt/splunk. 

[splunk@servername bin]$ echo $SPLUNK_HOME
/opt/splunk

So kindly let me know how to create a alias so that will do it and test the same.

 

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust
I suppose that you are using bash on that host.
Then just add to user splunk's (or what ever is your splunk user) ${HOME}/.bashrc the next line
alias splunk_ds = /opt/splunk/bin/splunk
then source it (source ~splunk/.bashrc) into shell or log in again to that user. After that you could use that command splunk_ds when you needs splunk in cli.
0 Karma

Vardhan
Contributor

Hi @anandhalagaras1 ,

 

What was your previous Splunk version?

0 Karma

anandhalagaras1
Path Finder

@Vardhan 

It is 7.2.1

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

This sounds like you have several splunk installation on your server and another installation has in your path instead of that in which directory you are.

r. Ismo

 

0 Karma

anandhalagaras1
Path Finder

@isoutamo ,

Thank for your reply. I have just checked in my /opt directory and  i can see two folders.

splunk 

splunkforwarder

 

So if i remove Splunkforwarder will the issue be sorted out. Kindly guide me.

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You should check how your organisation has defined to use UF on Splunk Server hosts. It could be that UFs are automatically installed all hosts and then it's not helping to remove it manually.

Best option is ensure that you are using full path /opt/splunk/bin/splunk all times when you are using those commands.

r. Ismo

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!