Couldn't able to perform any commands in Deployment Server

[anandhalagaras1]

Hi Team,

We have recently upgraded our Deployment Master server from 7.3.1 to 8.1.2 version. The upgrade seems to be success whereas if i perform any of the commands below I am getting an error. Refer below for more information.

Commands:

splunk restart

splunk start

splunk stop

splunk reload deploy-server

Errors:

[splunk@servername bin]$ splunk reload deploy-server
Error processing line 1 of /opt/splunk/lib/python3.7/site-packages/zc.lockfile-2.0-py3.7-nspkg.pth:

Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "/opt/splunk/lib/python3.7/importlib/util.py", line 14, in <module>
from contextlib import contextmanager
File "/opt/splunk/lib/python3.7/contextlib.py", line 5, in <module>
from collections import deque
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 579, in <module>
main()
File "/opt/splunk/lib/python3.7/site.py", line 566, in main
known_paths = addsitepackages(known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 349, in addsitepackages
addsitedir(sitedir, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 207, in addsitedir
addpackage(sitedir, name, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 178, in addpackage
import traceback
File "/opt/splunk/lib/python3.7/traceback.py", line 3, in <module>
import collections
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

So kindly help to check and let me know how to resolve the issue. But our Deployment server has been upgraded successfully to 8.1.2 but couldn't able to perform any commands due to this issue.

Whereas if we ran the command with (./) as prefix then i can able to execute all commands:

./splunk restart

./splunk start

./splunk stop

./splunk reload deploy-server

So kindly help me to fix this issue.

[Vardhan]

Hi @anandhalagaras1 ,

Yes remove the Splunkforwarder package and keep only the Splunk package.

In Linux, any Splunk command should run with (./) as a prefix. Whereas in windows you can run the Splunk commands without  (./) as a prefix.

[anandhalagaras1]

@Vardhan ,

I have removed the Splunk forwarder package from /opt and restarted the splunk services. But still I am getting the same python error.

i.e. Previously before upgrade I can able to ran the below commands  from any directory (without ./) and it could fetch the results.

splunk restart

splunk start

splunk reload deploy-server

But now Eventhough I have removed the splunk forwarder package from /opt I couldn't able to execute the command from any directory and especially without ./

So kindly guide me how to fix it.

[isoutamo]

What is output of: which splunk

[anandhalagaras1]

The output is pointing out to /opt/splunk.

[isoutamo]

Is it pointing to /opt/splunk/bin/splunk or /opt/splunk?

What you have in your $PATH?

Maybe you should create alias splunk_ds = /opt/splunk/bin/splunk and use it instead of splunk?

[anandhalagaras1]

@isoutamo 

If i ran the below command it is pointing to /opt/splunk. 

[splunk@servername bin]$ echo $SPLUNK_HOME
/opt/splunk

So kindly let me know how to create a alias so that will do it and test the same.

[isoutamo]

I suppose that you are using bash on that host.
Then just add to user splunk's (or what ever is your splunk user) ${HOME}/.bashrc the next line
alias splunk_ds = /opt/splunk/bin/splunk
then source it (source ~splunk/.bashrc) into shell or log in again to that user. After that you could use that command splunk_ds when you needs splunk in cli.

[Vardhan]

Hi @anandhalagaras1 ,

What was your previous Splunk version?

[anandhalagaras1]

@Vardhan 

It is 7.2.1

[isoutamo]

Hi

This sounds like you have several splunk installation on your server and another installation has in your path instead of that in which directory you are.

r. Ismo

[anandhalagaras1]

@isoutamo ,

Thank for your reply. I have just checked in my /opt directory and  i can see two folders.

splunk 

splunkforwarder

So if i remove Splunkforwarder will the issue be sorted out. Kindly guide me.

[isoutamo]

You should check how your organisation has defined to use UF on Splunk Server hosts. It could be that UFs are automatically installed all hosts and then it's not helping to remove it manually.

Best option is ensure that you are using full path /opt/splunk/bin/splunk all times when you are using those commands.

r. Ismo