Splunk doesn't launch

[njansons]

I was just trying to get splunk to run on port 443. I ran the following and then I get the error below.

$SPLUNK_HOME/opt/splunk/bin/splunk enable boot-start -user splunk

sudo setcap 'cap_net_bind_service=+ep' $SPLUNK_HOME/opt/splunk/bin/splunk
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk

sudo setcap 'cap_net_bind_service=+ep' $SPLUNK_HOME/opt/splunk/bin/splunkd
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd

ubuntu@hostname:/opt/splunk$ sudo $SPLUNK_HOME/opt/splunk/bin/splunk start
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Did not find "disabled" setting of "kvstore" stanza in server bundle.

Splunk> 4TW

Checking prerequisites...
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Checking mgmt port [8089]: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
open
Checking configuration... Done.
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Validating databases (splunkd validatedb) failed with code '127'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

ubuntu@hostname:/opt/splunk$ ls -altr lib bin
lib:
total 9020
-r--r--r-- 1 splunk splunk 247 Jan 31 19:44 copyright.txt
-r-xr-xr-x 1 splunk splunk 94120 Jan 31 20:08 libz.so.1.2.8
-r-xr-xr-x 1 splunk splunk 323616 Jan 31 20:08 libxslt.so.1.1.28
-r-xr-xr-x 1 splunk splunk 1817200 Jan 31 20:08 libxml2.so.2.9.2
-r-xr-xr-x 1 splunk splunk 966456 Jan 31 20:08 libsqlite3.so.0.8.6
-r-xr-xr-x 1 splunk splunk 272232 Jan 31 20:08 libpcre.so.1.2.5
-r-xr-xr-x 1 splunk splunk 193128 Jan 31 20:08 libmongoc-1.0.so.0.0.0
-r-xr-xr-x 1 splunk splunk 218624 Jan 31 20:08 libjemalloc.so.1
-r-xr-xr-x 1 splunk splunk 98296 Jan 31 20:08 libexslt.so.0.8.17
-r-xr-xr-x 1 splunk splunk 2867944 Jan 31 20:08 libcrypto.so.1.0.0
-r-xr-xr-x 1 splunk splunk 66296 Jan 31 20:08 libbz2.so.1.0.3
-r-xr-xr-x 1 splunk splunk 598912 Jan 31 20:08 libarchive.so.13.1.2
-r-xr-xr-x 1 splunk splunk 474824 Jan 31 20:08 libxmlsec1.so.1.2.20
-r-xr-xr-x 1 splunk splunk 314648 Jan 31 20:08 libxmlsec1-openssl.so.1.2.20
-r-xr-xr-x 1 splunk splunk 491656 Jan 31 20:08 libssl.so.1.0.0
-r-xr-xr-x 1 splunk splunk 203368 Jan 31 20:08 libmongoc-priv.so.0.0.0
-r-xr-xr-x 1 splunk splunk 156112 Jan 31 20:08 libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 13 Jan 31 20:13 libz.so -> libz.so.1.2.8
lrwxrwxrwx 1 splunk splunk 19 Jan 31 20:13 libsqlite3.so.0 -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 splunk splunk 23 Jan 31 20:13 libmongoc-priv.so.0 -> libmongoc-priv.so.0.0.0
lrwxrwxrwx 1 splunk splunk 22 Jan 31 20:13 libmongoc-1.0.so -> libmongoc-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libcrypto.so -> libcrypto.so.1.0.0
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libbz2.so -> libbz2.so.1.0.3
lrwxrwxrwx 1 splunk splunk 13 Jan 31 20:13 libz.so.1 -> libz.so.1.2.8
lrwxrwxrwx 1 splunk splunk 17 Jan 31 20:13 libxslt.so.1 -> libxslt.so.1.1.28
lrwxrwxrwx 1 splunk splunk 17 Jan 31 20:13 libxslt.so -> libxslt.so.1.1.28
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libxmlsec1.so.1 -> libxmlsec1.so.1.2.20
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libxmlsec1.so -> libxmlsec1.so.1.2.20
lrwxrwxrwx 1 splunk splunk 28 Jan 31 20:13 libxmlsec1-openssl.so.1 -> libxmlsec1-openssl.so.1.2.20
lrwxrwxrwx 1 splunk splunk 28 Jan 31 20:13 libxmlsec1-openssl.so -> libxmlsec1-openssl.so.1.2.20
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libxml2.so.2 -> libxml2.so.2.9.2
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libxml2.so -> libxml2.so.2.9.2
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libssl.so -> libssl.so.1.0.0
lrwxrwxrwx 1 splunk splunk 19 Jan 31 20:13 libsqlite3.so -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libpcre.so.1 -> libpcre.so.1.2.5
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libpcre.so -> libpcre.so.1.2.5
lrwxrwxrwx 1 splunk splunk 23 Jan 31 20:13 libmongoc-priv.so -> libmongoc-priv.so.0.0.0
lrwxrwxrwx 1 splunk splunk 22 Jan 31 20:13 libmongoc-1.0.so.0 -> libmongoc-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libjemalloc.so -> libjemalloc.so.1
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libexslt.so.0 -> libexslt.so.0.8.17
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libexslt.so -> libexslt.so.0.8.17
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libbz2.so.1 -> libbz2.so.1.0.3
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libbson-1.0.so.0 -> libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libbson-1.0.so -> libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libarchive.so.13 -> libarchive.so.13.1.2
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libarchive.so -> libarchive.so.13.1.2
drwxr-xr-x 18 splunk splunk 4096 Feb 18 06:09 node_modules
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 engines
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 with_stats
drwxr-xr-x 6 splunk splunk 4096 Feb 18 06:09 .
drwxr-xr-x 20 splunk splunk 20480 Feb 18 07:17 python2.7
drwxr-xr-x 9 splunk splunk 4096 Mar 24 02:51 ..

bin:
total 69624
-r-xr-xr-x 1 splunk splunk 3628 Jan 15 12:25 cherryd
-r-xr-xr-x 1 splunk splunk 1154 Jan 31 19:44 untarit.py
-r-xr-xr-x 1 splunk splunk 1060 Jan 31 19:44 tarit.py
-r--r--r-- 1 splunk splunk 1360 Jan 31 19:44 setSplunkEnv
-r-xr-xr-x 1 splunk splunk 7390 Jan 31 19:44 pid_check.sh
-r-xr-xr-x 1 splunk splunk 2191 Jan 31 19:44 installit.py
-r-xr-xr-x 1 splunk splunk 144 Jan 31 19:44 genWebCert.sh
-r-xr-xr-x 1 splunk splunk 205 Jan 31 19:44 genWebCert.py
-r-xr-xr-x 1 splunk splunk 206 Jan 31 19:44 genSignedServerCert.sh
-r-xr-xr-x 1 splunk splunk 211 Jan 31 19:44 genSignedServerCert.py
-r-xr-xr-x 1 splunk splunk 2367 Jan 31 19:44 genRootCA.sh
-r-xr-xr-x 1 splunk splunk 348 Jan 31 19:44 genAuditKeys.py
-r--r--r-- 1 splunk splunk 247 Jan 31 19:44 copyright.txt
-rwxr-xr-x 1 splunk splunk 3393 Jan 31 19:44 coldToFrozenExample.py
-r-xr-xr-x 1 splunk splunk 2477 Jan 31 19:44 tsidx_scan.py
-r-xr-xr-x 1 splunk splunk 2407 Jan 31 19:44 tocsv.py
-r-xr-xr-x 1 splunk splunk 23849 Jan 31 19:44 scrubber.py
-r-xr-xr-x 1 splunk splunk 5824 Jan 31 19:44 safe_restart_cluster_master.py
-r-xr-xr-x 1 splunk splunk 2759 Jan 31 19:44 runScript.py
-r-xr-xr-x 1 splunk splunk 2032 Jan 31 19:44 rest_handler.py
-r-xr-xr-x 1 splunk splunk 3753 Jan 31 19:44 parse_xml_buckets.py
-r-xr-xr-x 1 splunk splunk 19734 Jan 31 19:44 fill_summary_index.py
-r-xr-xr-x 1 splunk splunk 391 Jan 31 19:44 dbmanipulator.py
-r-xr-xr-x 1 splunk splunk 189536 Jan 31 20:08 tsidxprobe_plo
-r-xr-xr-x 1 splunk splunk 191776 Jan 31 20:08 tsidxprobe
-r-xr-xr-x 1 splunk splunk 201840 Jan 31 20:08 splunk-optimize
-r-xr-xr-x 1 splunk splunk 13496 Jan 31 20:08 splunkmon
-r-xr-xr-x 1 splunk splunk 28285264 Jan 31 20:08 splunkd
-r-xr-xr-x 1 splunk splunk 381136 Jan 31 20:08 splunk
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 signtool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 searchtest
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 recover-metadata
-r-xr-xr-x 1 splunk splunk 1779816 Jan 31 20:08 python2.7
-r-xr-xr-x 1 splunk splunk 2598800 Jan 31 20:08 pcregextest
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 parsetest
-r-xr-xr-x 1 splunk splunk 586480 Jan 31 20:08 openssl
-r-xr-xr-x 1 splunk splunk 9852664 Jan 31 20:08 node
-r-xr-xr-x 1 splunk splunk 23163840 Jan 31 20:08 mongod
-r-xr-xr-x 1 splunk splunk 2730416 Jan 31 20:08 locktool
-r-xr-xr-x 1 splunk splunk 186504 Jan 31 20:08 locktest
-r-xr-xr-x 1 splunk splunk 7576 Jan 31 20:08 jsmin
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 importtool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 exporttool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 classify
-r-xr-xr-x 1 splunk splunk 29952 Jan 31 20:08 bzip2
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 btprobe
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 btool
-r-xr-xr-x 1 splunk splunk 185040 Jan 31 20:08 walklex
-r-xr-xr-x 1 splunk splunk 21160 Jan 31 20:08 srm
-r-xr-xr-x 1 splunk splunk 200592 Jan 31 20:08 splunk-optimize-lex
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 bloom
lrwxrwxrwx 1 splunk splunk 9 Jan 31 20:13 python2 -> python2.7
lrwxrwxrwx 1 splunk splunk 9 Jan 31 20:13 python -> python2.7
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 scripts
drwxr-xr-x 3 splunk splunk 4096 Feb 18 06:09 jars
drwxr-xr-x 4 splunk splunk 4096 Feb 18 06:09 .
drwxr-xr-x 9 splunk splunk 4096 Mar 24 02:51 ..

[pcieniek]

This is not Splunk specific issue. You can find the answer here:

https://stackoverflow.com/questions/9843178/linux-capabilities-setcap-seems-to-disable-ld-library-pa...

Generally "LD_LIBRARY_PATH=/opt/splunk/lib" get's ignored after setcap for security reasons.

Workaround is via adding Splunk libraries globally, obviously with downsides of it own

$ sudo sh -c "echo \"/opt/splunk/lib\" >> /etc/ld.so.conf.d/splunk.conf"
$ sudo ldconfig

[petreb]

ldd /opt/splunk/bin/splunkd and see if all required lib dependencies are met;

[njansons]

Thanks petreb. Seems to be something wrong there. Output below. What happened? This was all working. Ideas on how to get it working again?

ubuntu@hostname:~$ ldd /opt/splunk/bin/splunkd
linux-vdso.so.1 => (0x00007ffe255ab000)
libjemalloc.so.1 => not found
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f40cea3b000)
libmongoc-1.0.so.0 => not found
libbson-1.0.so.0 => not found
libpcre.so.1 => not found
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f40ce6d4000)
libxslt.so.1 => not found
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f40ce475000)
libxmlsec1.so.1 => not found
libxmlsec1-openssl.so.1 => not found
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f40ce099000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f40cde95000)
libarchive.so.13 => not found
libbz2.so.1 => /lib/x86_64-linux-gnu/libbz2.so.1 (0x00007f40cdc85000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f40cd9cc000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f40cd7b3000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f40cd4ad000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f40cd28f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f40cceca000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f40ccca8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f40cec43000)

ubuntu@hostname:/opt/splunk/lib$ readlink -f libjemalloc.so
/opt/splunk/lib/libjemalloc.so.1
ubuntu@hostname:/opt/splunk/lib$ readlink -f libjemalloc.so.1
/opt/splunk/lib/libjemalloc.so.1

ubuntu@hostname:/opt/splunk/lib$ ls -altr libjemalloc.*
-r-xr-xr-x 1 splunk splunk 218624 Jan 31 20:08 libjemalloc.so.1
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libjemalloc.so -> libjemalloc.so.1

ubuntu@hostname:/opt/splunk/lib$ file libjemalloc.*
libjemalloc.so: symbolic link to `libjemalloc.so.1'
libjemalloc.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

ubuntu@hostname:/opt/splunk/lib$ uname -a
Linux hostname 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[swatghare]

Hi
Did you got solution to this problem,
I am having same issue.

Regards
Sushant

[chimell]

Log in with sudo su permission
Then go to /bin repertory
enter
./splunk start

Example
write
ubuntu@hostname: sudo su /opt/splunk/bin
on the following line type

./splunk start

[njansons]

Hi Chimell,

Thanks for the suggestion. This hasn't worked for me. It still has the same error.

Cheers,
NJ