topic Splunk forwarder on Linux - ./splunk "commands" just hang in Deployment Architecture

Splunk forwarder on Linux - ./splunk "commands" just hang

rune_hellem — Sun, 08 Mar 2020 18:13:23 GMT

It has been a while since I have worked with Linux, but doing my best to refresh my knowledge. Successfully installed the latest forwarder on Ubuntu and it has actually phoned home and the deployment server has pushed config to it. But now it has stopped working. I have configured it to run as user 'splunk', not as root. This has caused some issue, for instance when I just now did run

splunk@myserver:~$ ./bin/splunk display deploy-client
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Operation "ospath_fopen" failed in /opt/splunk/src/libzero/conf-mutator-locking.c:337, conf_mutator_lock(); Permission denied

Did sudo to root and
/opt/splunkforwarder# chown -R splunk:splunk *
Error did go away, but now when running the same command (as Splunk) nothing happens. I must CTRL+C to "get out of it"

splunk@myserver:~$ ./bin/splunk display deploy-client
^C
splunk@myserver:~$

Most likely more a basic Linux quesiton, but still, anyone who has an idea of what could be wrong?

Update
And now I did try

splunk@myserver:~$ ./bin/splunk list forward-server
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied

Since I did do the chown, this should not happen, so quite sure that I've done something not totally correct when installing as root and then switching to Splunk as described here https://docs.splunk.com/Documentation/Splunk/8.0.2/Admin/ConfigureSplunktostartatboottime#Enable_boot-start_as_a_non-root_user - well, it is simply just the chown-command, but since

splunk@myserver:~$ ls -la /opt/splunkforwarder/etc/apps/learned/metadata/local.meta
-rw------- 1 root root 531 Mar  8 19:15 /opt/splunkforwarder/etc/apps/learned/metadata/local.meta

Something is not correct on the server.

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

richgalloway — Sun, 08 Mar 2020 20:41:59 GMT

Did you stop Splunk before running chown?

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

manjunathmeti — Sun, 08 Mar 2020 20:43:25 GMT

Splunk service might be running under root user. First as splunk user try to stop service. If this doesn't work, then stop it as sudo or root user.

/opt/splunkforwarder/bin/splunk stop

Then Change home path permissions to splunk:splunk.

chown -R splunk:splunk /opt/splunkforwarder

Then start splunk and check permissions of the files.

/opt/splunkforwarder/bin/splunk start

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

PavelP — Sun, 08 Mar 2020 20:48:03 GMT

the fact, that you get permission denied error after you ran chown indicate some other process modified the files afterwards, this could be just a running as root splunk process. You must frist stop spunk and then run the chown command.

But... scratch that! Instead of tinker with linux permissions do it right from the beginning: the only command(*) you need to execute as root during splunk administration is

splunk enable boot-start -user splunk

optionally with "-systemd-managed 1" if you run it on a systemd-enabled system.
After that, you have to use

su - splunk

to switch to splunk user (or similar command) and work as a restricted splunk user only.

Why not login as splunk user directly from the start? Many splunkd admins do so, but it is better to leave this account without OS password so nobody except root can login. By doing so you effectively reducing the attack surface.

(*) - of course you need modify file permissions, disable THP, adjust limits etc as root before and during you work with splunk, but it is linux admin operation, not splunk administration 🙂

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

rune_hellem — Mon, 09 Mar 2020 09:55:20 GMT

Did check that, if I try to stop/start as user Splunk and permissions are wrong it will fail. I then sudo to root, change the owership and retry as user Splunk. It will then manage to stop and start. But still
splunk display deploy-client
will just hang.

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

rune_hellem — Mon, 09 Mar 2020 11:25:34 GMT

I did a complete uninstall, and then instead of

sudo su
dpkg -i /tmp/splunkforwarder-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb

I did

sudo dpkg -i /tmp/splunkforwarder-8.0.2-a7f645ddaf91-linux-2.6-amd64.deb

Result was not that messy, but still

splunk@myserver:~/bin$ ./splunk  list forward-server

And it still hangs

And btw - I do not know the password for the Splunk-user, so I have to

sudo su splunk

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

rune_hellem — Mon, 09 Mar 2020 13:23:55 GMT

I have not yet found the answer to my question, but I am quite sure that I have nailed it down to the cli not showing the prompt for the username/password - as it does whenever I try the same commands on Windows boxes where we have installed the forwarder.

Therefore i have created a new question with more precise information.

Re: Splunk forwarder on Linux - ./splunk "commands" just hang

PavelP — Tue, 10 Mar 2020 11:55:33 GMT

"complete uninstall" can still leave some files in /opt/splunkforwarder. Try this:

sudo /etc/init.d/SplunkForwarder stop # OR systemctl stop SplunkForwarder
ps aux|grep -i splunk # to be 100% sure there are no splunk processes running
sudo rm -r /opt/splunkforwarder # or any other folder where you've had it installed
sudo apt-get install /tmp/splunkforwarder*deb
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 1
systemctl start SplunkForwarder
sudo su - splunk
/opt/splunkforwarder/bin/splunk add forward-server <splunk>:9997 # or use deployment server / deploy an app
/opt/splunkforwarder/bin/splunk list forward-server

let us know if it worked