Knowledge Management

Why is there an error when trying to start Splunk first time?

sombhtr239
Explorer

We tried to install splunk 8.1.0 and after untarring the file tried to start splunk both as root and splunk user via /opt/splunk/bin/splunk start 

Error comes up as execve: Operation not permitted while running command /opt/splunk/bin/splunkd

Any urgent help is appreciated

Labels (1)
Tags (2)
0 Karma
1 Solution

sombhtr239
Explorer

I found this issue more with the server. Every time we reboot the server, it failed to turn on. Finally created a new VM which resolve the problem. Thanks all for your contribution. Issue resolved now.

View solution in original post

0 Karma

Lamech
New Member

For others who may encounter this problem in the future. You will see this error on RHEL 8 and derivatives if you are using the tar.gz installation and fapolicyd is enabled and you have not whitelisted the splunk installation directory

To confirm whether fapolicyd is preventing you from running splunk you can use the following audit log search command

sudo ausearch --start today -m fanotify -i

you will see output like this:

node=rhel8.example.com type=PROCTITLE msg=audit(03/14/2022 17:43:30.150:21294716) : proctitle=/opt/splunk/bin/splunk start
node=rhel8.example.com type=PATH msg=audit(03/14/2022 17:43:30.150:21294716) : item=0 name=/opt/splunk/bin/python3.7 inode=4328145 dev=fd:03 mode=file,555 ouid=splunk ogid=splunk rdev=00:00 obj=unconfined_u:object_r:usr_t:s0 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
node=rhel8.example.com type=CWD msg=audit(03/14/2022 17:43:30.150:21294716) : cwd=/home/lamech
node=rhel8.example.com type=SYSCALL msg=audit(03/14/2022 17:43:30.150:21294716) : arch=x86_64 syscall=execve success=no exit=EPERM(Operation not permitted) a0=0x5604c24f1980 a1=0x7ffeb0c870d0 a2=0x7ffeb0c86f40 a3=0x5 items=1 ppid=228305 pid=228318 auid=lamech uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=3 comm=splunk exe=/opt/splunk/bin/splunk subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
node=rhel8.example.com type=FANOTIFY msg=audit(03/14/2022 17:43:30.150:21294716) : resp=deny
0 Karma

xNGreenex
Engager

I just ran into this same issue.  

 

To whitelist the application ran the following:

fapolicyd-cli --file add /opt/splunk

fapolicyd-cli --update

systemctl restart fapolicyd

 

Jamie
Explorer

If you just did the install, any reason why you didn't use 8.2.3?

I know you say you have tried as splunk, does that mean you created the account and then did su - splunk?

Show the output of

 ls -ld /opt/splunk

and

ls -l /opt/splunk

After you have tried, show the output of

tail -30 /var/log/messages 

 

0 Karma

sombhtr239
Explorer

We are trying to scale up our environment and the other 2 SH peer are running on SE8.0.5, hence tried both 8.0.5 and 8.1.0.  Will check /var/log/messages; however already changed permission for /opt/splunk . Tried to run via systemctl as well to No go as of now.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

When Splunk is run as root, a number of files are written with root as the owner.  Trying to run Splunk as a different user will fail until those files are given to the different user.

chown -r splunk:splunk /opt/splunk

You may have to use systemctl to start splunk.

sudo systemctl start splunk
---
If this reply helps you, Karma would be appreciated.
0 Karma

sombhtr239
Explorer

Tried steps already, its still not working. We tried the following to no go:

1) Tried to start splunk as splunk user did not go

2) Kept filesystem with Splunk and tried to start splunk with root did not work.

3) Tried to enable systemctl did not work.

0 Karma

sombhtr239
Explorer

I found this issue more with the server. Every time we reboot the server, it failed to turn on. Finally created a new VM which resolve the problem. Thanks all for your contribution. Issue resolved now.

0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...