topic Re: Couldn't able to perform any commands in Deployment Server in Splunk Search

Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Mon, 29 Mar 2021 05:31:36 GMT

Hi Team,

We have recently upgraded our Deployment Master server from 7.3.1 to 8.1.2 version. The upgrade seems to be success whereas if i perform any of the commands below I am getting an error. Refer below for more information.

Commands:

splunk restart

splunk start

splunk stop

splunk reload deploy-server

Errors:

[splunk@servername bin]$ splunk reload deploy-server
Error processing line 1 of /opt/splunk/lib/python3.7/site-packages/zc.lockfile-2.0-py3.7-nspkg.pth:

Fatal Python error: initsite: Failed to import the site module
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 168, in addpackage
exec(line)
File "<string>", line 1, in <module>
File "/opt/splunk/lib/python3.7/importlib/util.py", line 14, in <module>
from contextlib import contextmanager
File "/opt/splunk/lib/python3.7/contextlib.py", line 5, in <module>
from collections import deque
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site.py", line 579, in <module>
main()
File "/opt/splunk/lib/python3.7/site.py", line 566, in main
known_paths = addsitepackages(known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 349, in addsitepackages
addsitedir(sitedir, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 207, in addsitedir
addpackage(sitedir, name, known_paths)
File "/opt/splunk/lib/python3.7/site.py", line 178, in addpackage
import traceback
File "/opt/splunk/lib/python3.7/traceback.py", line 3, in <module>
import collections
File "/opt/splunk/lib/python3.7/collections/__init__.py", line 27, in <module>
from reprlib import recursive_repr as _recursive_repr
File "/opt/splunk/lib/python2.7/site-packages/reprlib/__init__.py", line 7, in <module>
raise ImportError('This package should not be accessible on Python 3. '
ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

So kindly help to check and let me know how to resolve the issue. But our Deployment server has been upgraded successfully to 8.1.2 but couldn't able to perform any commands due to this issue.

Whereas if we ran the command with (./) as prefix then i can able to execute all commands:

./splunk restart

./splunk start

./splunk stop

./splunk reload deploy-server

So kindly help me to fix this issue.

Re: Couldn't able to perform any commands in Deployment Server

isoutamo — Mon, 29 Mar 2021 06:42:50 GMT

This sounds like you have several splunk installation on your server and another installation has in your path instead of that in which directory you are.

r. Ismo

Re: Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Mon, 29 Mar 2021 07:21:44 GMT

@isoutamo ,

Thank for your reply. I have just checked in my /opt directory and i can see two folders.

splunk

splunkforwarder

So if i remove Splunkforwarder will the issue be sorted out. Kindly guide me.

Re: Couldn't able to perform any commands in Deployment Server

Vardhan — Mon, 29 Mar 2021 08:50:43 GMT

Hi @anandhalagaras1 ,

Yes remove the Splunkforwarder package and keep only the Splunk package.

In Linux, any Splunk command should run with (./) as a prefix. Whereas in windows you can run the Splunk commands without (./) as a prefix.

Re: Couldn't able to perform any commands in Deployment Server

isoutamo — Mon, 29 Mar 2021 09:08:20 GMT

You should check how your organisation has defined to use UF on Splunk Server hosts. It could be that UFs are automatically installed all hosts and then it's not helping to remove it manually.

Best option is ensure that you are using full path /opt/splunk/bin/splunk all times when you are using those commands.

r. Ismo

Re: Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Mon, 29 Mar 2021 14:14:17 GMT

@Vardhan ,

I have removed the Splunk forwarder package from /opt and restarted the splunk services. But still I am getting the same python error.

i.e. Previously before upgrade I can able to ran the below commands from any directory (without ./) and it could fetch the results.

splunk restart

splunk start

splunk reload deploy-server

But now Eventhough I have removed the splunk forwarder package from /opt I couldn't able to execute the command from any directory and especially without ./

So kindly guide me how to fix it.

Re: Couldn't able to perform any commands in Deployment Server

Vardhan — Mon, 29 Mar 2021 14:52:17 GMT

Hi @anandhalagaras1 ,

What was your previous Splunk version?

Re: Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Mon, 29 Mar 2021 16:18:03 GMT

@Vardhan

It is 7.2.1

Re: Couldn't able to perform any commands in Deployment Server

isoutamo — Tue, 30 Mar 2021 08:42:23 GMT

What is output of: which splunk

Re: Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Tue, 30 Mar 2021 06:43:02 GMT

The output is pointing out to /opt/splunk.

Re: Couldn't able to perform any commands in Deployment Server

isoutamo — Tue, 30 Mar 2021 08:58:39 GMT

Is it pointing to /opt/splunk/bin/splunk or /opt/splunk?

What you have in your $PATH?

Maybe you should create alias splunk_ds = /opt/splunk/bin/splunk and use it instead of splunk?

Re: Couldn't able to perform any commands in Deployment Server

anandhalagaras1 — Tue, 30 Mar 2021 10:01:03 GMT

@isoutamo

If i ran the below command it is pointing to /opt/splunk.

[splunk@servername bin]$ echo $SPLUNK_HOME
/opt/splunk

So kindly let me know how to create a alias so that will do it and test the same.

Re: Couldn't able to perform any commands in Deployment Server

isoutamo — Thu, 01 Apr 2021 05:16:07 GMT

I suppose that you are using bash on that host.
Then just add to user splunk's (or what ever is your splunk user) ${HOME}/.bashrc the next line
alias splunk_ds = /opt/splunk/bin/splunk
then source it (source ~splunk/.bashrc) into shell or log in again to that user. After that you could use that command splunk_ds when you needs splunk in cli.