Why am I getting an installation failure for Splunk 6.2.2, 6.2.3, and 6.2.4 in OS X 10.11 beta and how do I fix this?

[danrand]

Now that the beta period is over, I can ask this question here. I have been attempting to install Splunk Enterprise (6.2.2, 6.2.3, 6.2.4) into OS X 10.11, but it fails every time with the following error (see image). The number in square brackets is usually different, but the failure mode is always the same - it attempts to check the kvstore port, finds something wrong, asks for a change (that would be invalid) and exits.

Checking prerequisites...
dyld: Library not loaded: /Users/eserv/wrangler/build-home/6.2.2/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/Splunk/bin/splunkd
  Reason: image not found
    Checking mgmt port [8089]: dyld: Library not loaded: /Users/eserv/wrangler/build-home/6.2.2/lib/libmongoc-1.0.0.dylib
  Referenced from: /Applications/Splunk/bin/splunkd
  Reason: image not found
open
    Checking kvstore port [1048584]: invalid; must be between 1 and 65535
Would you like to change ports? [y/n]: n
Exiting....

Is there an explanation and a fix for this?

[sloshburch]

FYI. The Splunk issue ID for this issue is EST-8128. Feel free to reference that when you look at release notes to see what's fixed in future versions. As always, there's the usual no guarantee on when this gets implemented (targeted release). I wanted to at least let folks know it was being tracked and not ignored.

[sloshburch]

FYI: Release notes for today's 6.3.3 show this issue as resolved.
http://docs.splunk.com/Documentation/Splunk/6.3.3/ReleaseNotes/6.3.3
SPL-110878, SPL-104699 Splunk does not start on OSX 10.11 (El Capitan).

Give it a shot (I have not yet).

[dmr195]

I can confirm that no special actions are required to get Splunk 6.3.3 to find its dynamic libraries on OS X 10.11 (El Capitan). I can see you've used the @executable_path and @loader_path linker variables to make it relocatable too. Many thanks for fixing this in an update rather than making everyone wait for the next major release.

There is just one tiny glitch, which is that if you install on a Mac that doesn't have the Apple Developer Tools installed then part way through the installation you get a popup that says, "The "DeRez" command requires the command line developer tools. Do you want to install them now?" However, if you click "Not Now" on this popup then everything still appears to work fine.

If you install on a Mac that has the Apple Developer Tools installed then everything is slick.

[instrumental]

Installed on 10.11 without any xcode. Hit the DeRez requires dev tools dialogue.

Hoping to avoid installing xcode and more heft/cruft/config, but also reluctant to fake the path as per the above solution as it means my notes on recovering the solution become pages long.

Found an alternate temporary solution; just Install the command line tools http://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/

Hopefully Splunk will find it!

[sloshburch]

Thanks for that feedback. I've just posted it back into the ticket info.

[triest]

The posted answers attempt to workaround correct this issue by making the libraries appear where Splunk is looking for them; IMHO the "correct" solution would be to have Splunk look in the "correct" location.

Note: I am not knocking the other solutions; they've been posted for months while I ignored the problem and something good enough today is better then something perfect tomorrow. I'm just trying to explain why I am posting a new answer.

Edit: Please see the comments below about rpath; that would be the "right" way to fix it. I'll try to update my answer utilizing rpath.

Solution

Run the following command (copy and paste into a terminal -- Applications -> Utilities -> Terminal)

splunk_home="/Applications/Splunk" ; for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do f=$(echo $file | rev | cut -d / -f 1 | rev) ; sudo install_name_tool -change $file $splunk_home/lib/$f $lib ; done ; done

Caveats

1. Assumes Splunk is installed in /Applications/Splunk (if not update splunk_home="..." )
2. Assumes you use an sh style shell (you probably do; if in doubt copy the script below into a file)
3. Only tested on two versions of Splunk Enterprise; they were different enough to require slight modifications so I think I've made it generic enough. If not post the version (and product if not Splunk Enterprise) and I'll try to update the answer

What does it do?

If you run otool -L <object>, it will print the shared libraries (e.g. otool -L /Applications/Splunk/bin/splunkd)

For Splunk 6.3.1 this gives us:

/Applications/Splunk/bin/splunkd:
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libmongoc-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libbson-1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.5.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxml2.2.dylib (compatibility version 12.0.0, current version 12.2.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.28.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1.1.dylib (compatibility version 4.0.0, current version 4.20.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libxmlsec1-openssl.1.dylib (compatibility version 4.0.0, current version 4.20.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libarchive.13.dylib (compatibility version 15.0.0, current version 15.2.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libbz2.1.dylib (compatibility version 2.0.0, current version 2.3.0)
    /Users/eserv/wrangler-2.0/build-home/ember/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
    /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

The last two libraries look good; the previous entries are looking for things in /Users/eserv/wrangler-2.0/build-home/ember/lib/ which probably doesn't exist
Note: the exact path is version specific; 6.2.3 was trying to use /Users/eserv/wrangler/build-home/6.2.3/lib/

For each library, you then need to run install_name_tool to correct the path:

install_name_tool -change /Users/eserv/wrangler/build-home/6.2.3/lib/libmongoc-1.0.0.dylib /Applications/Splunk/lib/libmongoc-1.0.0.dylib /Applications/Splunk/bin/splunkd

After running similar commands 12 times, you'd try to start Splunk only to discover /Applications/Splunk/lib/libmongoc-1.0.0.dylib references /Users/eserv/wrangler/build-home/6.2.3/lib/libbson-1.0.0.dylib. To put an end to the rabbit hole, I just looped over all libraries in /Applications/Splunk/lib

At this point, my laziness kicked in and I wrote a script:

#!/bin/sh

splunk_home="/Applications/Splunk"

for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do  
  for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do
    f=$(echo $file | rev | cut -d / -f 1 | rev)
    sudo install_name_tool -change $file $splunk_home/lib/$f $lib
  done
done

Save it to a file, make it executable, and execute it.

Note: The command at the top of this post is just a minimized version of this script

[dmr195]

A couple of extra points to add to this answer are:

• It only works if you have the Apple developer tools installed on your Mac. Without the developer tools install_name_tool and otool just print a message saying you need to install the developer tools:

  xcode-select: error: no developer tools were found at '/Applications/Xcode.app', and no install could be requested (perhaps no UI is present), please install manually from 'developer.apple.com'.

• You can make it even more generic by replacing $splunk_home/lib/$f with "@rpath/$f" on the install_name_tool command line, plus adding a call to install_name_tool -add_rpath "@loader_path/../lib" just once for each file. In other words, change the script at the end of the answer to:

  !/bin/sh

  splunk_home="/Applications/splunk"

  for lib in $splunk_home/bin/splunkd $(ls $splunk_home/lib/*.dylib); do
  sudo install_name_tool -add_rpath "@loader_path/../lib" $lib
  for file in $(otool -L $lib | grep "/Users" | cut -d " " -f 1); do
  f=$(echo $file | rev | cut -d / -f 1 | rev)
  sudo install_name_tool -change $file "@rpath/$f" $lib
  done
  done

Using @rpath and @Anonymous_path means that the correct directories will be searched even if Splunk is relocated. This is the professional way to do it and I would very much hope that the Splunk dev team will be incorporating something like this into their build process in the near future!

[jrodman]

Huh, we use rpath already on some platforms. No real reason we don't use it on os x.

[sloshburch]

Heads up: I did the install_name_tool solution but it left me with some buggy behavior while developing a modular input. Once I switched to the symlink, the buggy behavior went bye bye.

I was so deep in the dev that I didn't think to post it here. I totally forget what the issue was (but will edit this post if I remember).

Since I'm not providing much substance, I don't want to sound like I'm knocking/preferring one approach over another. I'd say the only takeaway is to keep in mind that if you see something odd - check if on a non mac system or try the sym link before burning a full day thinking you made a mistake...not that I was that foolish...of course I was 🙂

[sklass]

Perhaps we could get an updated release from Splunk folks which fixes the problem..

[Lowell]

Really great answer! Thanks for posting it. I love the use of install_name_tool over symlinking. I'm assuming this will allow me to run multiple versions of Splunk side-by-side (in different install folders, of course) which is something I need to do from time-to-time.

[martinstack]

Thank you so much for this!

[sklass]

+1 - Nice work

[ryanoconnor]

$ mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
$ cd /Users/eserv/wrangler-2.0/build-home/ember
$ ln -s /Applications/splunk/lib lib

This worked on Splunk 6.3 and OS X 10.11 El Capitan

[triddell]

Here were the exact commands I used. Change "tim" to your main username below:

sudo mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
sudo chown -R tim:staff /Users/eserv/
cd /Users/eserv/wrangler-2.0/build-home/ember
ln -s /Applications/splunk/lib lib

This was Splunk 6.3 and El Capitan too.

[gbeddow]

Ran across a similar problem trying to install Splunk Lite under the final version of OS X 10.11 El Capitan.

The instructions from yulhos were almost correct – something like this worked for me:

$ mkdir -p /Users/eserv/wrangler-2.0/build-home/6.3.0
$ cd /Users/eserv/wrangler-2.0/build-home/6.3.0
$ ln -s /Applications/splunk/lib lib
$ mkdir -p /Users/eserv/wrangler-2.0/build-home/ember
$ cd /Users/eserv/wrangler-2.0/build-home/ember
$ ln -s /Applications/splunk/lib lib

Then reinstalled from the .dmg and restarted Splunk:

$ /Applications/Splunk/bin/splunk start

and the web interface came up at:

http://127.0.0.1:8000

[joewiz]

Worked for me with OS X 10.11 El Capitan, except I had to prepend sudo before each command.

[hazclan13]

Helped me too, thanks!

[rsolutions]

This worked for me.. just changed the path to /opt/splunk/lib as I did customize my install path.

[danrand]

After making the dir and the symlink I installed 6.3 from dmg. OS X complained a few times and I had to rm and recreate the symlink but I finally got it to work.
Tl;dr it's a little messy but it does solve the problem and allows splunk to work again.