Why is there an error when trying to start Splunk first time?

[sombhtr239]

We tried to install splunk 8.1.0 and after untarring the file tried to start splunk both as root and splunk user via /opt/splunk/bin/splunk start 

Error comes up as execve: Operation not permitted while running command /opt/splunk/bin/splunkd

Any urgent help is appreciated

[Lamech]

For others who may encounter this problem in the future. You will see this error on RHEL 8 and derivatives if you are using the tar.gz installation and fapolicyd is enabled and you have not whitelisted the splunk installation directory

To confirm whether fapolicyd is preventing you from running splunk you can use the following audit log search command

sudo ausearch --start today -m fanotify -i

you will see output like this:

node=rhel8.example.com type=PROCTITLE msg=audit(03/14/2022 17:43:30.150:21294716) : proctitle=/opt/splunk/bin/splunk start
node=rhel8.example.com type=PATH msg=audit(03/14/2022 17:43:30.150:21294716) : item=0 name=/opt/splunk/bin/python3.7 inode=4328145 dev=fd:03 mode=file,555 ouid=splunk ogid=splunk rdev=00:00 obj=unconfined_u:object_r:usr_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=rhel8.example.com type=CWD msg=audit(03/14/2022 17:43:30.150:21294716) : cwd=/home/lamech
node=rhel8.example.com type=SYSCALL msg=audit(03/14/2022 17:43:30.150:21294716) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x5604c24f1980 a1=0x7ffeb0c870d0 a2=0x7ffeb0c86f40 a3=0x5 items=1 ppid=228305 pid=228318 auid=lamech uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=3 comm=splunk exe=/opt/splunk/bin/splunk subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
node=rhel8.example.com type=FANOTIFY msg=audit(03/14/2022 17:43:30.150:21294716) : resp=deny

[xNGreenex]

I just ran into this same issue.  

To whitelist the application ran the following:

fapolicyd-cli --file add /opt/splunk

fapolicyd-cli --update

systemctl restart fapolicyd

[Jamie]

If you just did the install, any reason why you didn't use 8.2.3?

I know you say you have tried as splunk, does that mean you created the account and then did su - splunk?

Show the output of

 ls -ld /opt/splunk

and

ls -l /opt/splunk

After you have tried, show the output of

tail -30 /var/log/messages 

[sombhtr239]

We are trying to scale up our environment and the other 2 SH peer are running on SE8.0.5, hence tried both 8.0.5 and 8.1.0.  Will check /var/log/messages; however already changed permission for /opt/splunk . Tried to run via systemctl as well to No go as of now.

[richgalloway]

When Splunk is run as root, a number of files are written with root as the owner.  Trying to run Splunk as a different user will fail until those files are given to the different user.

chown -r splunk:splunk /opt/splunk

You may have to use systemctl to start splunk.

sudo systemctl start splunk

[sombhtr239]

Tried steps already, its still not working. We tried the following to no go:

1) Tried to start splunk as splunk user did not go

2) Kept filesystem with Splunk and tried to start splunk with root did not work.

3) Tried to enable systemctl did not work.

[sombhtr239 · Accepted solution]

I found this issue more with the server. Every time we reboot the server, it failed to turn on. Finally created a new VM which resolve the problem. Thanks all for your contribution. Issue resolved now.