How to resolve [Errno 13] Permission denied?

[julian0125]

Hello friend,

I've got the next issue trying to run ./splunk start or status. How can i fix it? i think it is a user permission issue.

[root@cerr500810 bin]# ./splunk start

Warning: cannot create "/monitoreo/splunk/var/log/splunk"

Warning: cannot create "/monitoreo/splunk/var/log/introspection"
Pid file "/monitoreo/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/Splunk_TA_nix/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/TA-tripwire_enterprise/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/lookup_editor/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/Splunk_TA_nix/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/TA-tripwire_enterprise/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/lookup_editor/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/Splunk_TA_nix/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/TA-tripwire_enterprise/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/lookup_editor/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /monitoreo/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/monitoreo/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/monitoreo/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

[arjunpkishore5]

I see that you already are root. This is bad practice. Never use root to start the service.

Next, check the permissions on the file system as suggested by @woodcock

If the permissions are set up correctly, check if the disk is mounted properly. I have seen instances where the disk is set to read-only mode accidentally by the linux admin. This makes startup or usage impossible since splunk will be unable to create files.

[aamer86]

I see Splunk is not installed under the default directory
try running this

you have to make sure Splunk is running from
/opt/splunk/bin/splunk start

1- change the splunk home directory to /opt/splunk
2- Run (using the root user) /opt/splunk/bin/splunk stop
3- chown -R splunk /opt/
4- sudo su splunk
5- /opt/splunk/bin/splunk start

[arjunpkishore5]

Hi @aamer86

This is incorrect. Default directory is just that - a default directory and not a mandatory directory. It's possible to change the base directory and have splunk running by updating the value of $SPLUNK_HOME

[woodcock]

This happens when you have started splunk as user root and then later try to start it as the correct non- root user (usually splunk). To fix, do this:

AS USER root:

/opt/splunk/bin/splunk start
chown -R splunk: $SPLUNK_HOME
service splunk start

[nnimbe1]

@woodcock this was the same issue which i was facing...and now using your steps it resolved thanks a lot

[andhika_pratama]

you might be root, but the Splunk owner is another user.
1. command ls -l or ll to know who the owner is
2. switch to Splunk user

[somesoni2]

Try using this command instead

sudo service splunk restart

It seems the splunk is running with different user and you're trying to restart it with user root. The above service command will ensure Splunk service will restart with the account it's setup to run with.

[julian0125]

hello @somesoni2

i tried with the command "sudo service splunk restart" but it shows the next:

[root@cerr500810 bin]# sudo service splunk restart
Redirecting to /bin/systemctl restart splunk.service
Job for splunk.service failed because the control process exited with error code. See "systemctl status splunk.service" and "journalctl -xe" for details.
[root@cerr500810 bin]# sudo service splunk restart
Redirecting to /bin/systemctl restart splunk.service
Job for splunk.service failed because the control process exited with error code. See "systemctl status splunk.service" and "journalctl -xe" for details.
[root@cerr500810 bin]# sudo service splunk start
Redirecting to /bin/systemctl start splunk.service
Job for splunk.service failed because the control process exited with error code. See "systemctl status splunk.service" and "journalctl -xe" for details.
[root@cerr500810 bin]# sudo service splunk status
Redirecting to /bin/systemctl status splunk.service
● splunk.service - splunk Service , para monitoreo de Seguridad
Loaded: loaded (/etc/systemd/system/splunk.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-02-21 20:28:16 UTC; 4s ago
Process: 24782 ExecStart=/usr/local/sbin/splunk.sh (code=exited, status=2)
Main PID: 24782 (code=exited, status=2)

Feb 21 20:28:15 cerr500810 systemd[1]: Starting splunk Service , para monitoreo de Seguridad...
Feb 21 20:28:16 cerr500810 systemd[1]: splunk.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 21 20:28:16 cerr500810 systemd[1]: Failed to start splunk Service , para monitoreo de Seguridad.
Feb 21 20:28:16 cerr500810 systemd[1]: Unit splunk.service entered failed state.
Feb 21 20:28:16 cerr500810 systemd[1]: splunk.service failed.

[somesoni2]

Can you run this and see under what user Splunkd service is running?

ps -ef  | grep splunkd | grep start | grep -v grep

[julian0125]

Sure! i've got this

[segemer@cerr500810 system]$ ps -ef | grep splunkd | grep start | grep -v grep
root 7805 1 0 Feb13 ? 00:29:16 splunkd -p 8089 start
root 7824 7805 0 Feb13 ? 00:00:00 [splunkd pid=7805] splunkd -p 8089 start [process-runner]
[segemer@cerr500810 system]$

[somesoni2]

Looks like at some point of time, Splunk started with wrong user. It's currently running as root. Do you always run Splunk as root OR use a non-root splunk user account?

Also, who owns Splunk file system (run ls -ltr /monitoreo/splunk)?
What's content of attribute SPLUNK_OS_USER in file /monitoreo/splunk/etc/splunk-launch.conf ?