Splunk initially connects to OpenLDAP server, but why are users unable to log in with error "Can't contact LDAP server"?

[jlbark]

I have a Splunk server that is trying to authenticate users via SSL to an LDAP server. I followed this documentation pretty closely:

I set the logging level to DEBUG so I can see what is happening under the covers. It looks like SPLUNK is able to initially contact the LDAP server and do some user caching:

...
09-25-2015 15:18:35.576 -0400 DEBUG AuthenticationManagerLDAP - Attempting to get user information for user="blah" from strategy="myldap"
09-25-2015 15:18:35.576 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Initializing with LDAPURL="ldaps://ldap1.mydomain.org:636"
09-25-2015 15:18:35.576 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Attempting bind as DN="cn=admin,dc=mydomain,dc=org"
09-25-2015 15:18:35.593 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Bind successful

09-25-2015 15:18:35.595 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Loading entry attributes for DN="uid=blah,ou=people,dc=mydomain,dc=org"
09-25-2015 15:18:35.595 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Adding attribute="cn" with value="Blah Blahski"
09-25-2015 15:18:35.595 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Adding attribute="uid" with value="blah"
09-25-2015 15:18:35.595 -0400 DEBUG AuthenticationManagerLDAP - Attempting to get roles for user="blah" with DN="uid=blah,ou=people,dc=mydomain,dc=org" in strategy="myldap"
09-25-2015 15:18:35.595 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Attempting to search subtree at DN="ou=group,dc=mydomain,dc=org" using filter="(&(memberuid=blah)(cn=*))"
09-25-2015 15:18:35.596 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Search duration="807 microseconds"
09-25-2015 15:18:35.596 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Loading entry attributes for DN="cn=ccrlog,ou=group,dc=mydomain,dc=org"
09-25-2015 15:18:35.596 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Adding attribute="cn" with value="loggers"
09-25-2015 15:18:35.596 -0400 DEBUG AuthenticationManagerLDAP - Mapping groups for user="blah" for group DN="cn=loggers,ou=group,dc=mydomain,dc=org"
09-25-2015 15:18:35.596 -0400 DEBUG AuthenticationManagerLDAP - Found matching group="loggers" with mapped roles
09-25-2015 15:18:35.596 -0400 DEBUG AuthenticationManagerLDAP - Successfully filled info for user="blah" with realname="Blah Blahski" and email="" in strategy="myldap"    

It continues like this for some time with no errors, it maps the groups to splunk roles perfectly. But then it tries to bind with the users credentials:

...
09-25-2015 15:18:35.596 -0400 DEBUG AuthenticationManagerLDAP - Caching user="blah" with DN="uid=blah,ou=people,dc=mydomain,dc=org"
09-25-2015 15:18:35.596 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Initializing with LDAPURL="ldaps://ldap1.mydomain.org:636"
09-25-2015 15:18:35.596 -0400 DEBUG ScopedLDAPConnection - strategy="myldap" Attempting bind as DN="uid=blah,ou=people,dc=mydomain,dc=org"
09-25-2015 15:18:35.598 -0400 ERROR ScopedLDAPConnection - strategy="myldap" Error binding to LDAP. reason="Can't contact LDAP server"
...

I know the users password is correct, because I can ssh to any LDAP enabled host with that password, and it works just fine. I know splunk is finding the user, because it is caching it:

09-25-2015 15:29:06.466 -0400 DEBUG AuthenticationManagerLDAP - Listing cached user="blah"

I can run the recomended tests, and they work just fine with the users credentials:

ldapsearch -h ldaps://ldap1.mydomain.org -p 636 -b ou=group,dc=mydomain,dc=org -x -D uid=blah,ou=people,dc=mydomain,dc=org -W "groupNameAttribute"

Information:

Splunk Version: 6.2.1
Operating System: RHEL 6.6
LDAP: OpenLDAP 2.4.39

Configurations:
1. $SPLUNK/etc/openldap/ldap.conf

  TLS_CACERTDIR $SPLUNK/etc/openldap/cacerts
  SASL_NOCANON    off
  URI ldaps://ldap1.mydomain.org ldaps://ldap2.mydomain.org
  BASE dc=mydomain,dc=org

1. $SPLUNK/etc/system/local/authentication.conf

   [authentication]
   authSettings = myldap
   authType = LDAP

     [myldap]
     SSLEnabled = 1
     anonymous_referrals = 1
     bindDN = cn=admin,dc=mydomain,dc=org
     bindDNpassword = $1$RandomHash
     charset = utf8
     emailAttribute = mail
     groupBaseDN = ou=group,dc=mydomain,dc=org
     groupMappingAttribute = uid
     groupMemberAttribute = memberuid
     groupNameAttribute = cn
     host = ldap1.mydomain.org
     nestedGroups = 0
     network_timeout = 20 
     port = 636
    realNameAttribute = cn
    sizelimit = 1000
    timelimit = 15
    userBaseDN = ou=people,dc=mydomain,dc=org
    userNameAttribute = uid
   
    [roleMap_myldap]
    user = loggers
   

   Any help would be greatly appreciated.

[jterry]

ok, so my problem turned out to be that the osx-bundled version of openldap was unnecessarily involving kerberos. i ended up brew-installing a vanilla/newer version & it works fine.

my config files are below:

authentication.conf:

[test_LDAP]
SSLEnabled = 0
anonymous_referrals = 1
bindDN = cn=Manager,dc=splunk,dc=com
bindDNpassword = secret
charset = utf8
emailAttribute = mail
groupBaseDN = ou=groups,dc=splunk,dc=com
groupMappingAttribute = dn
groupMemberAttribute = member
groupNameAttribute = cn
host = <YOUR_IP_HERE>
nestedGroups = 0
network_timeout = 20
port = 389
realNameAttribute = cn
sizelimit = 1000
timelimit = 15
userBaseDN = ou=users,dc=splunk,dc=com
userNameAttribute = cn

[authentication]
authSettings = test_LDAP
authType = LDAP

[roleMap_test_LDAP]
admin = admin
power = power
user = user

ldif:

#
# create the organization
#
dn: dc=splunk,dc=com
objectClass: dcObject
objectClass: organization
o: splunk

#
# create the group branch
#
dn: ou=groups,dc=splunk,dc=com
objectClass: organizationalunit
ou: groups
description: generic groups branch

#
# create the users branch
#
dn: ou=users,dc=splunk,dc=com
objectClass: organizationalunit
ou: users
description: generic users branch

#
# create user1
#
dn: cn=user1,ou=users,dc=splunk,dc=com
objectClass: top
objectClass: person
cn: user1
sn: user1
userPassword: {SSHA}JfbsnOFacwlwSjH2IgtGqcgGXqJUfYvR

#
# create user2
#
dn: cn=user2,ou=users,dc=splunk,dc=com
objectClass: top
objectClass: person
cn: user2
sn: user2
userPassword: {SSHA}SNkjwFLo+e4BVGTMZqx4q5cR51dIQ++b

#
# create power1
#
dn: cn=power1,ou=users,dc=splunk,dc=com
objectClass: top
objectClass: person
cn: power1
sn: power1
userPassword: {SSHA}7X2tcL+AZSG4FYhYptkfQF7QMejLSUxj

#
# create admin1
#
dn: cn=admin1,ou=users,dc=splunk,dc=com
objectClass: top
objectClass: person
cn: admin1
sn: admin1
userPassword: {SSHA}L9x9nnPYdFnbNzHr8jaBWFNIjrk3h2Cs

#
# create the regular users group
#
dn: cn=user,ou=groups,dc=splunk,dc=com
objectClass: groupofnames
cn: user
description: the regular users group
member: cn=user1,ou=users,dc=splunk,dc=com
member: cn=user2,ou=users,dc=splunk,dc=com

#
# create the power users group
#
dn: cn=power,ou=groups,dc=splunk,dc=com
objectClass: groupofnames
cn: power
description: the power users group
member: cn=power1,ou=users,dc=splunk,dc=com

#
# create the admin users group
#
dn: cn=admin,ou=groups,dc=splunk,dc=com
objectClass: groupofnames
cn: admin
description: the admin users group
member: cn=admin1,ou=users,dc=splunk,dc=com

slapd.conf:

#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include     /usr/local/etc/openldap/schema/core.schema

# Define global ACLs to disable default read access.

# Do not enable referrals until AFTER you have a working directory
# service AND an understanding of referrals.
#referral   ldap://root.openldap.org

pidfile     /usr/local/var/run/slapd.pid
argsfile    /usr/local/var/run/slapd.args

# Load dynamic backend modules:
modulepath  /usr/local/Cellar/openldap/2.4.41_1/libexec/openldap
moduleload  back_bdb.la
moduleload  back_hdb.la
moduleload  back_ldap.la
moduleload  back_mdb.la

# Sample security restrictions
#   Require integrity protection (prevent hijacking)
#   Require 112-bit (3DES or better) encryption for updates
#   Require 63-bit encryption for simple bind
# security ssf=1 update_ssf=112 simple_bind=64

# Sample access control policy:
#   Root DSE: allow anyone to read it
#   Subschema (sub)entry DSE: allow anyone to read it
#   Other DSEs:
#       Allow self write access
#       Allow authenticated users read access
#       Allow anonymous users to authenticate
#   Directives needed to implement policy:
# access to dn.base="" by * read
# access to dn.base="cn=Subschema" by * read
# access to *
#   by self write
#   by users read
#   by anonymous auth
#
# if no access controls are present, the default policy
# allows anyone and everyone to read anything but restricts
# updates to rootdn.  (e.g., "access to * by * read")
#
# rootdn can always read and write EVERYTHING!

#######################################################################
# BDB database definitions
#######################################################################

#database   bdb
database    mdb
suffix      "dc=splunk,dc=com"
rootdn      "cn=Manager,dc=splunk,dc=com"
# Cleartext passwords, especially for the rootdn, should
# be avoid.  See slappasswd(8) and slapd.conf(5) for details.
# Use of strong authentication encouraged.
#rootpw     secret
rootpw      {SSHA}y3IN0Qc7ajG7vHo4ymMcZSG59Yivy4Jn
# The database directory MUST exist prior to running slapd AND 
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory   /usr/local/var/openldap-data
# Indices to maintain
index   objectClass eq

good luck,
j

[jterry]

i'm not using kerberos either. You might try removing SSL from the equation 1st. to narrow the scope a bit.

[jlbark]

Thanks for the Tip... I hope this helps someone. I am not using Kerberos though.

[jlbark]

Also I think one of the main differences is that I am using SSL.

[jterry]

i'm working on this problem as well. i'll post back if i find anything useful.

[jlbark]

Wow...Glad I could get some help on this...LOL....I guess Splunk answers is not very active.