Installation

Why am I getting an installation failure for Splunk 6.2.2, 6.2.3, and 6.2.4 in OS X 10.11 beta and how do I fix this?

danrand
Explorer

Now that the beta period is over, I can ask this question here. I have been attempting to install Splunk Enterprise (6.2.2, 6.2.3, 6.2.4) into OS X 10.11, but it fails every time with the following error (see image). The number in square brackets is usually different, but the failure mode is always the same - it attempts to check the kvstore port, finds something wrong, asks for a change (that would be invalid) and exits.

Checking prerequisites...
dyld: Library not loaded: /Users/eserv/wrangler/build-home/6.2.2/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/Splunk/bin/splunkd
  Reason: image not found
    Checking mgmt port [8089]: dyld: Library not loaded: /Users/eserv/wrangler/build-home/6.2.2/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/Splunk/bin/splunkd
  Reason: image not found
open
    Checking kvstore port [1048584]: invalid; must be between 1 and 65535
Would you like to change ports? [y/n]: n
Exiting....

Is there an explanation and a fix for this?

Labels (1)
1 Solution

yulhos
Engager

I think DEV Team use static library path for mongodb.

I fix it like below:

1) mkdir -p /Users/eserv/wrangler/build-home/6.2.6
2) cd /Users/eserv/wrangler/build-home/6.2.6
3) ln -s /Applications/splunk/lib lib

View solution in original post

sloshburch
Splunk Employee
Splunk Employee

FYI. The Splunk issue ID for this issue is EST-8128. Feel free to reference that when you look at release notes to see what's fixed in future versions. As always, there's the usual no guarantee on when this gets implemented (targeted release). I wanted to at least let folks know it was being tracked and not ignored.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

FYI: Release notes for today's 6.3.3 show this issue as resolved.
http://docs.splunk.com/Documentation/Splunk/6.3.3/ReleaseNotes/6.3.3
SPL-110878, SPL-104699 Splunk does not start on OSX 10.11 (El Capitan).

Give it a shot (I have not yet).

0 Karma

dmr195
Communicator

I can confirm that no special actions are required to get Splunk 6.3.3 to find its dynamic libraries on OS X 10.11 (El Capitan). I can see you've used the @executable_path and @loader_path linker variables to make it relocatable too. Many thanks for fixing this in an update rather than making everyone wait for the next major release.

There is just one tiny glitch, which is that if you install on a Mac that doesn't have the Apple Developer Tools installed then part way through the installation you get a popup that says, "The "DeRez" command requires the command line developer tools. Do you want to install them now?" However, if you click "Not Now" on this popup then everything still appears to work fine.

If you install on a Mac that has the Apple Developer Tools installed then everything is slick.

0 Karma

instrumental
New Member

Installed on 10.11 without any xcode. Hit the DeRez requires dev tools dialogue.

Hoping to avoid installing xcode and more heft/cruft/config, but also reluctant to fake the path as per the above solution as it means my notes on recovering the solution become pages long.

Found an alternate temporary solution; just Install the command line tools http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/

Hopefully Splunk will find it!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Thanks for that feedback. I've just posted it back into the ticket info.

0 Karma

triest
Communicator

The posted answers attempt to workaround correct this issue by making the libraries appear where Splunk is looking for them; IMHO the "correct" solution would be to have Splunk look in the "correct" location.

Note: I am not knocking the other solutions; they've been posted for months while I ignored the problem and something good enough today is better then something perfect tomorrow. I'm just trying to explain why I am posting a new answer.

Edit: Please see the comments below about rpath; that would be the "right" way to fix it. I'll try to update my answer utilizing rpath.

Solution

Run the following command (copy and paste into a terminal -- Applications -> Utilities -> Terminal)

splunk_home="/Applications/Splunk" ; for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do f=$(echo $file | rev | cut -d / -f 1 | rev) ; sudo install_name_tool -change $file $splunk_home/lib/$f $lib ; done ; done

Caveats

  1. Assumes Splunk is installed in /Applications/Splunk (if not update splunk_home="..." )
  2. Assumes you use an sh style shell (you probably do; if in doubt copy the script below into a file)
  3. Only tested on two versions of Splunk Enterprise; they were different enough to require slight modifications so I think I've made it generic enough. If not post the version (and product if not Splunk Enterprise) and I'll try to update the answer

What does it do?

If you run otool -L <object>, it will print the shared libraries (e.g. otool -L /Applications/Splunk/bin/splunkd)

For Splunk 6.3.1 this gives us:

/Applications/Splunk/bin/splunkd:
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libbson-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.5.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.2.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.28.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1.1.dylib (compatibility version 4.0.0, current version 4.20.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1-openssl.1.dylib (compatibility version 4.0.0, current version 4.20.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libarchive.13.dylib (compatibility version 15.0.0, current version 15.2.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libbz2.1.dylib (compatibility version 2.0.0, current version 2.3.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

The last two libraries look good; the previous entries are looking for things in /Users/eserv/wrangler-2.0/build-home/ember/lib/ which probably doesn't exist
Note: the exact path is version specific; 6.2.3 was trying to use /Users/eserv/wrangler/build-home/6.2.3/lib/

For each library, you then need to run install_name_tool to correct the path:

install_name_tool -change /Users/eserv/wrangler/build-home/6.2.3/lib/libmongoc-1.0.0.dylib /Applications/Splunk/lib/libmongoc-1.0.0.dylib /Applications/Splunk/bin/splunkd

After running similar commands 12 times, you'd try to start Splunk only to discover /Applications/Splunk/lib/libmongoc-1.0.0.dylib references /Users/eserv/wrangler/build-home/6.2.3/lib/libbson-1.0.0.dylib. To put an end to the rabbit hole, I just looped over all libraries in /Applications/Splunk/lib

At this point, my laziness kicked in and I wrote a script:

#!/bin/sh

splunk_home="/Applications/Splunk"

for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do  
  for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do
    f=$(echo $file | rev | cut -d / -f 1 | rev)
    sudo install_name_tool -change $file $splunk_home/lib/$f $lib
  done
done

Save it to a file, make it executable, and execute it.

Note: The command at the top of this post is just a minimized version of this script

dmr195
Communicator

A couple of extra points to add to this answer are:

  • It only works if you have the Apple developer tools installed on your Mac. Without the developer tools install_name_tool and otool just print a message saying you need to install the developer tools:

    xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.

  • You can make it even more generic by replacing $splunk_home/lib/$f with "@rpath/$f" on the install_name_tool command line, plus adding a call to install_name_tool -add_rpath "@loader_path/../lib" just once for each file. In other words, change the script at the end of the answer to:

    !/bin/sh

    splunk_home="/Applications/splunk"

    for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do
    sudo install_name_tool -add_rpath "@loader_path/../lib" $lib
    for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do
    f=$(echo $file | rev | cut -d / -f 1 | rev)
    sudo install_name_tool -change $file "@rpath/$f" $lib
    done
    done

Using @rpath and @loader_path means that the correct directories will be searched even if Splunk is relocated. This is the professional way to do it and I would very much hope that the Splunk dev team will be incorporating something like this into their build process in the near future!

0 Karma

jrodman
Splunk Employee
Splunk Employee

Huh, we use rpath already on some platforms. No real reason we don't use it on os x.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Heads up: I did the install_name_tool solution but it left me with some buggy behavior while developing a modular input. Once I switched to the symlink, the buggy behavior went bye bye.

I was so deep in the dev that I didn't think to post it here. I totally forget what the issue was (but will edit this post if I remember).

Since I'm not providing much substance, I don't want to sound like I'm knocking/preferring one approach over another. I'd say the only takeaway is to keep in mind that if you see something odd - check if on a non mac system or try the sym link before burning a full day thinking you made a mistake...not that I was that foolish...of course I was 🙂

0 Karma

sklass
Path Finder

Perhaps we could get an updated release from Splunk folks which fixes the problem..

0 Karma

Lowell
Super Champion

Really great answer! Thanks for posting it. I love the use of install_name_tool over symlinking. I'm assuming this will allow me to run multiple versions of Splunk side-by-side (in different install folders, of course) which is something I need to do from time-to-time.

martinstack
New Member

Thank you so much for this!

0 Karma

sklass
Path Finder

+1 - Nice work

0 Karma

ryanoconnor
Builder

$ mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
$ cd /Users/eserv/wrangler-2.0/build-home/ember
$ ln -s /Applications/splunk/lib lib

This worked on Splunk 6.3 and OS X 10.11 El Capitan

triddell
Engager

Here were the exact commands I used. Change "tim" to your main username below:

sudo mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
sudo chown -R tim:staff /Users/eserv/
cd /Users/eserv/wrangler-2.0/build-home/ember
ln -s /Applications/splunk/lib lib

This was Splunk 6.3 and El Capitan too.

gbeddow
Explorer

Ran across a similar problem trying to install Splunk Lite under the final version of OS X 10.11 El Capitan.

The instructions from yulhos were almost correct – something like this worked for me:

$ mkdir -p /Users/eserv/wrangler-2.0/build-home/6.3.0
$ cd /Users/eserv/wrangler-2.0/build-home/6.3.0
$ ln -s /Applications/splunk/lib lib
$ mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
$ cd /Users/eserv/wrangler-2.0/build-home/ember
$ ln -s /Applications/splunk/lib lib

Then reinstalled from the .dmg and restarted Splunk:

$ /Applications/Splunk/bin/splunk start

and the web interface came up at:

http://127.0.0.1:8000

joewiz
Engager

Worked for me with OS X 10.11 El Capitan, except I had to prepend sudo before each command.

0 Karma

hazclan13
Engager

Helped me too, thanks!

0 Karma

rsolutions
Path Finder

This worked for me.. just changed the path to /opt/splunk/lib as I did customize my install path.

0 Karma

danrand
Explorer

After making the dir and the symlink I installed 6.3 from dmg. OS X complained a few times and I had to rm and recreate the symlink but I finally got it to work.
Tl;dr it's a little messy but it does solve the problem and allows splunk to work again.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...