topic Re: Unable to start splunk first time in Knowledge Management

Why is there an error when trying to start Splunk first time?

sombhtr239 — Mon, 06 Jun 2022 17:40:01 GMT

We tried to install splunk 8.1.0 and after untarring the file tried to start splunk both as root and splunk user via /opt/splunk/bin/splunk start

Error comes up as execve: Operation not permitted while running command /opt/splunk/bin/splunkd

Any urgent help is appreciated

Re: Unable to start splunk first time

richgalloway — Thu, 11 Nov 2021 13:59:43 GMT

When Splunk is run as root, a number of files are written with root as the owner. Trying to run Splunk as a different user will fail until those files are given to the different user.

chown -r splunk:splunk /opt/splunk

You may have to use systemctl to start splunk.

sudo systemctl start splunk

Re: Unable to start splunk first time

sombhtr239 — Thu, 11 Nov 2021 14:34:04 GMT

Tried steps already, its still not working. We tried the following to no go:

1) Tried to start splunk as splunk user did not go

2) Kept filesystem with Splunk and tried to start splunk with root did not work.

3) Tried to enable systemctl did not work.

Re: Unable to start splunk first time

Jamie — Thu, 11 Nov 2021 16:45:16 GMT

If you just did the install, any reason why you didn't use 8.2.3?

I know you say you have tried as splunk, does that mean you created the account and then did su - splunk?

Show the output of

 ls -ld /opt/splunk

and

ls -l /opt/splunk

After you have tried, show the output of

tail -30 /var/log/messages

Re: Unable to start splunk first time

sombhtr239 — Fri, 12 Nov 2021 07:49:02 GMT

We are trying to scale up our environment and the other 2 SH peer are running on SE8.0.5, hence tried both 8.0.5 and 8.1.0. Will check /var/log/messages; however already changed permission for /opt/splunk . Tried to run via systemctl as well to No go as of now.

Re: Unable to start splunk first time

sombhtr239 — Mon, 15 Nov 2021 11:09:20 GMT

I found this issue more with the server. Every time we reboot the server, it failed to turn on. Finally created a new VM which resolve the problem. Thanks all for your contribution. Issue resolved now.

Re: Unable to start splunk first time

Lamech — Mon, 14 Mar 2022 23:32:37 GMT

For others who may encounter this problem in the future. You will see this error on RHEL 8 and derivatives if you are using the tar.gz installation and fapolicyd is enabled and you have not whitelisted the splunk installation directory

To confirm whether fapolicyd is preventing you from running splunk you can use the following audit log search command

sudo ausearch --start today -m fanotify -i

you will see output like this:

node=rhel8.example.com type=PROCTITLE msg=audit(03/14/2022 17:43:30.150:21294716) : proctitle=/opt/splunk/bin/splunk start node=rhel8.example.com type=PATH msg=audit(03/14/2022 17:43:30.150:21294716) : item=0 name=/opt/splunk/bin/python3.7 inode=4328145 dev=fd:03 mode=file,555 ouid=splunk ogid=splunk rdev=00:00 obj=unconfined_u:object_r:usr_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 node=rhel8.example.com type=CWD msg=audit(03/14/2022 17:43:30.150:21294716) : cwd=/home/lamech node=rhel8.example.com type=SYSCALL msg=audit(03/14/2022 17:43:30.150:21294716) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x5604c24f1980 a1=0x7ffeb0c870d0 a2=0x7ffeb0c86f40 a3=0x5 items=1 ppid=228305 pid=228318 auid=lamech uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=3 comm=splunk exe=/opt/splunk/bin/splunk subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null) node=rhel8.example.com type=FANOTIFY msg=audit(03/14/2022 17:43:30.150:21294716) : resp=deny

Re: Unable to start splunk first time

xNGreenex — Mon, 06 Jun 2022 17:25:01 GMT

I just ran into this same issue.

To whitelist the application ran the following:

fapolicyd-cli --file add /opt/splunk

fapolicyd-cli --update

systemctl restart fapolicyd