Monitoring Splunk

Splunk doesn't launch

njansons
Explorer

I was just trying to get splunk to run on port 443. I ran the following and then I get the error below.

$SPLUNK_HOME/opt/splunk/bin/splunk enable boot-start -user splunk

sudo setcap 'cap_net_bind_service=+ep' $SPLUNK_HOME/opt/splunk/bin/splunk
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunk

sudo setcap 'cap_net_bind_service=+ep' $SPLUNK_HOME/opt/splunk/bin/splunkd
sudo setcap 'cap_net_bind_service=+ep' /opt/splunk/bin/splunkd

ubuntu@hostname:/opt/splunk$ sudo $SPLUNK_HOME/opt/splunk/bin/splunk start
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Did not find "disabled" setting of "kvstore" stanza in server bundle.

Splunk> 4TW

Checking prerequisites...
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Checking mgmt port [8089]: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
open
Checking configuration... Done.
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory
Validating databases (splunkd validatedb) failed with code '127'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

ubuntu@hostname:/opt/splunk$ ls -altr lib bin
lib:
total 9020
-r--r--r-- 1 splunk splunk 247 Jan 31 19:44 copyright.txt
-r-xr-xr-x 1 splunk splunk 94120 Jan 31 20:08 libz.so.1.2.8
-r-xr-xr-x 1 splunk splunk 323616 Jan 31 20:08 libxslt.so.1.1.28
-r-xr-xr-x 1 splunk splunk 1817200 Jan 31 20:08 libxml2.so.2.9.2
-r-xr-xr-x 1 splunk splunk 966456 Jan 31 20:08 libsqlite3.so.0.8.6
-r-xr-xr-x 1 splunk splunk 272232 Jan 31 20:08 libpcre.so.1.2.5
-r-xr-xr-x 1 splunk splunk 193128 Jan 31 20:08 libmongoc-1.0.so.0.0.0
-r-xr-xr-x 1 splunk splunk 218624 Jan 31 20:08 libjemalloc.so.1
-r-xr-xr-x 1 splunk splunk 98296 Jan 31 20:08 libexslt.so.0.8.17
-r-xr-xr-x 1 splunk splunk 2867944 Jan 31 20:08 libcrypto.so.1.0.0
-r-xr-xr-x 1 splunk splunk 66296 Jan 31 20:08 libbz2.so.1.0.3
-r-xr-xr-x 1 splunk splunk 598912 Jan 31 20:08 libarchive.so.13.1.2
-r-xr-xr-x 1 splunk splunk 474824 Jan 31 20:08 libxmlsec1.so.1.2.20
-r-xr-xr-x 1 splunk splunk 314648 Jan 31 20:08 libxmlsec1-openssl.so.1.2.20
-r-xr-xr-x 1 splunk splunk 491656 Jan 31 20:08 libssl.so.1.0.0
-r-xr-xr-x 1 splunk splunk 203368 Jan 31 20:08 libmongoc-priv.so.0.0.0
-r-xr-xr-x 1 splunk splunk 156112 Jan 31 20:08 libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 13 Jan 31 20:13 libz.so -> libz.so.1.2.8
lrwxrwxrwx 1 splunk splunk 19 Jan 31 20:13 libsqlite3.so.0 -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 splunk splunk 23 Jan 31 20:13 libmongoc-priv.so.0 -> libmongoc-priv.so.0.0.0
lrwxrwxrwx 1 splunk splunk 22 Jan 31 20:13 libmongoc-1.0.so -> libmongoc-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libcrypto.so -> libcrypto.so.1.0.0
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libbz2.so -> libbz2.so.1.0.3
lrwxrwxrwx 1 splunk splunk 13 Jan 31 20:13 libz.so.1 -> libz.so.1.2.8
lrwxrwxrwx 1 splunk splunk 17 Jan 31 20:13 libxslt.so.1 -> libxslt.so.1.1.28
lrwxrwxrwx 1 splunk splunk 17 Jan 31 20:13 libxslt.so -> libxslt.so.1.1.28
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libxmlsec1.so.1 -> libxmlsec1.so.1.2.20
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libxmlsec1.so -> libxmlsec1.so.1.2.20
lrwxrwxrwx 1 splunk splunk 28 Jan 31 20:13 libxmlsec1-openssl.so.1 -> libxmlsec1-openssl.so.1.2.20
lrwxrwxrwx 1 splunk splunk 28 Jan 31 20:13 libxmlsec1-openssl.so -> libxmlsec1-openssl.so.1.2.20
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libxml2.so.2 -> libxml2.so.2.9.2
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libxml2.so -> libxml2.so.2.9.2
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libssl.so -> libssl.so.1.0.0
lrwxrwxrwx 1 splunk splunk 19 Jan 31 20:13 libsqlite3.so -> libsqlite3.so.0.8.6
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libpcre.so.1 -> libpcre.so.1.2.5
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libpcre.so -> libpcre.so.1.2.5
lrwxrwxrwx 1 splunk splunk 23 Jan 31 20:13 libmongoc-priv.so -> libmongoc-priv.so.0.0.0
lrwxrwxrwx 1 splunk splunk 22 Jan 31 20:13 libmongoc-1.0.so.0 -> libmongoc-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libjemalloc.so -> libjemalloc.so.1
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libexslt.so.0 -> libexslt.so.0.8.17
lrwxrwxrwx 1 splunk splunk 18 Jan 31 20:13 libexslt.so -> libexslt.so.0.8.17
lrwxrwxrwx 1 splunk splunk 15 Jan 31 20:13 libbz2.so.1 -> libbz2.so.1.0.3
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libbson-1.0.so.0 -> libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libbson-1.0.so -> libbson-1.0.so.0.0.0
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libarchive.so.13 -> libarchive.so.13.1.2
lrwxrwxrwx 1 splunk splunk 20 Jan 31 20:13 libarchive.so -> libarchive.so.13.1.2
drwxr-xr-x 18 splunk splunk 4096 Feb 18 06:09 node_modules
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 engines
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 with_stats
drwxr-xr-x 6 splunk splunk 4096 Feb 18 06:09 .
drwxr-xr-x 20 splunk splunk 20480 Feb 18 07:17 python2.7
drwxr-xr-x 9 splunk splunk 4096 Mar 24 02:51 ..

bin:
total 69624
-r-xr-xr-x 1 splunk splunk 3628 Jan 15 12:25 cherryd
-r-xr-xr-x 1 splunk splunk 1154 Jan 31 19:44 untarit.py
-r-xr-xr-x 1 splunk splunk 1060 Jan 31 19:44 tarit.py
-r--r--r-- 1 splunk splunk 1360 Jan 31 19:44 setSplunkEnv
-r-xr-xr-x 1 splunk splunk 7390 Jan 31 19:44 pid_check.sh
-r-xr-xr-x 1 splunk splunk 2191 Jan 31 19:44 installit.py
-r-xr-xr-x 1 splunk splunk 144 Jan 31 19:44 genWebCert.sh
-r-xr-xr-x 1 splunk splunk 205 Jan 31 19:44 genWebCert.py
-r-xr-xr-x 1 splunk splunk 206 Jan 31 19:44 genSignedServerCert.sh
-r-xr-xr-x 1 splunk splunk 211 Jan 31 19:44 genSignedServerCert.py
-r-xr-xr-x 1 splunk splunk 2367 Jan 31 19:44 genRootCA.sh
-r-xr-xr-x 1 splunk splunk 348 Jan 31 19:44 genAuditKeys.py
-r--r--r-- 1 splunk splunk 247 Jan 31 19:44 copyright.txt
-rwxr-xr-x 1 splunk splunk 3393 Jan 31 19:44 coldToFrozenExample.py
-r-xr-xr-x 1 splunk splunk 2477 Jan 31 19:44 tsidx_scan.py
-r-xr-xr-x 1 splunk splunk 2407 Jan 31 19:44 tocsv.py
-r-xr-xr-x 1 splunk splunk 23849 Jan 31 19:44 scrubber.py
-r-xr-xr-x 1 splunk splunk 5824 Jan 31 19:44 safe_restart_cluster_master.py
-r-xr-xr-x 1 splunk splunk 2759 Jan 31 19:44 runScript.py
-r-xr-xr-x 1 splunk splunk 2032 Jan 31 19:44 rest_handler.py
-r-xr-xr-x 1 splunk splunk 3753 Jan 31 19:44 parse_xml_buckets.py
-r-xr-xr-x 1 splunk splunk 19734 Jan 31 19:44 fill_summary_index.py
-r-xr-xr-x 1 splunk splunk 391 Jan 31 19:44 dbmanipulator.py
-r-xr-xr-x 1 splunk splunk 189536 Jan 31 20:08 tsidxprobe_plo
-r-xr-xr-x 1 splunk splunk 191776 Jan 31 20:08 tsidxprobe
-r-xr-xr-x 1 splunk splunk 201840 Jan 31 20:08 splunk-optimize
-r-xr-xr-x 1 splunk splunk 13496 Jan 31 20:08 splunkmon
-r-xr-xr-x 1 splunk splunk 28285264 Jan 31 20:08 splunkd
-r-xr-xr-x 1 splunk splunk 381136 Jan 31 20:08 splunk
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 signtool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 searchtest
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 recover-metadata
-r-xr-xr-x 1 splunk splunk 1779816 Jan 31 20:08 python2.7
-r-xr-xr-x 1 splunk splunk 2598800 Jan 31 20:08 pcregextest
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 parsetest
-r-xr-xr-x 1 splunk splunk 586480 Jan 31 20:08 openssl
-r-xr-xr-x 1 splunk splunk 9852664 Jan 31 20:08 node
-r-xr-xr-x 1 splunk splunk 23163840 Jan 31 20:08 mongod
-r-xr-xr-x 1 splunk splunk 2730416 Jan 31 20:08 locktool
-r-xr-xr-x 1 splunk splunk 186504 Jan 31 20:08 locktest
-r-xr-xr-x 1 splunk splunk 7576 Jan 31 20:08 jsmin
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 importtool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 exporttool
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 classify
-r-xr-xr-x 1 splunk splunk 29952 Jan 31 20:08 bzip2
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 btprobe
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 btool
-r-xr-xr-x 1 splunk splunk 185040 Jan 31 20:08 walklex
-r-xr-xr-x 1 splunk splunk 21160 Jan 31 20:08 srm
-r-xr-xr-x 1 splunk splunk 200592 Jan 31 20:08 splunk-optimize-lex
-r-xr-xr-x 1 splunk splunk 48360 Jan 31 20:08 bloom
lrwxrwxrwx 1 splunk splunk 9 Jan 31 20:13 python2 -> python2.7
lrwxrwxrwx 1 splunk splunk 9 Jan 31 20:13 python -> python2.7
drwxr-xr-x 2 splunk splunk 4096 Feb 18 06:09 scripts
drwxr-xr-x 3 splunk splunk 4096 Feb 18 06:09 jars
drwxr-xr-x 4 splunk splunk 4096 Feb 18 06:09 .
drwxr-xr-x 9 splunk splunk 4096 Mar 24 02:51 ..

Tags (1)

pcieniek
Loves-to-Learn Lots

This is not Splunk specific issue. You can find the answer here:

https://stackoverflow.com/questions/9843178/linux-capabilities-setcap-seems-to-disable-ld-library-pa...

Generally "LD_LIBRARY_PATH=/opt/splunk/lib" get's ignored after setcap for security reasons.

 

Workaround is via adding Splunk libraries globally, obviously with downsides of it own

 

$ sudo sh -c "echo \"/opt/splunk/lib\" >> /etc/ld.so.conf.d/splunk.conf"
$ sudo ldconfig

 

 

0 Karma

petreb
Path Finder

ldd /opt/splunk/bin/splunkd and see if all required lib dependencies are met;

0 Karma

njansons
Explorer

Thanks petreb. Seems to be something wrong there. Output below. What happened? This was all working. Ideas on how to get it working again?

ubuntu@hostname:~$ ldd /opt/splunk/bin/splunkd
linux-vdso.so.1 => (0x00007ffe255ab000)
libjemalloc.so.1 => not found
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f40cea3b000)
libmongoc-1.0.so.0 => not found
libbson-1.0.so.0 => not found
libpcre.so.1 => not found
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f40ce6d4000)
libxslt.so.1 => not found
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f40ce475000)
libxmlsec1.so.1 => not found
libxmlsec1-openssl.so.1 => not found
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f40ce099000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f40cde95000)
libarchive.so.13 => not found
libbz2.so.1 => /lib/x86_64-linux-gnu/libbz2.so.1 (0x00007f40cdc85000)
libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f40cd9cc000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f40cd7b3000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f40cd4ad000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f40cd28f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f40cceca000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f40ccca8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f40cec43000)

ubuntu@hostname:/opt/splunk/lib$ readlink -f libjemalloc.so
/opt/splunk/lib/libjemalloc.so.1
ubuntu@hostname:/opt/splunk/lib$ readlink -f libjemalloc.so.1
/opt/splunk/lib/libjemalloc.so.1

ubuntu@hostname:/opt/splunk/lib$ ls -altr libjemalloc.*
-r-xr-xr-x 1 splunk splunk 218624 Jan 31 20:08 libjemalloc.so.1
lrwxrwxrwx 1 splunk splunk 16 Jan 31 20:13 libjemalloc.so -> libjemalloc.so.1

ubuntu@hostname:/opt/splunk/lib$ file libjemalloc.*
libjemalloc.so: symbolic link to `libjemalloc.so.1'
libjemalloc.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

ubuntu@hostname:/opt/splunk/lib$ uname -a
Linux hostname 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

0 Karma

swatghare
Path Finder

Hi
Did you got solution to this problem,
I am having same issue.

Regards
Sushant

0 Karma

chimell
Motivator

Log in with sudo su permission
Then go to /bin repertory
enter
./splunk start

Example
write
ubuntu@hostname: sudo su /opt/splunk/bin
on the following line type

./splunk start
0 Karma

njansons
Explorer

Hi Chimell,

Thanks for the suggestion. This hasn't worked for me. It still has the same error.

Cheers,
NJ

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...