How to resolve 'ldapsearch' error ?

[damode]

Whenever I run any ldap query on the Search Head, I get results, however, I always get the below error,

[INDEXER] External search command 'ldapsearch' returned error code 1. Script output = " ERROR "KeyError at ""C:\Program Files\Splunk\var\run\searchpeers\SEARCH-HEAD-1511928556\apps\SA-ldapsearch\bin\packages\splunklib\client.py"", line 1653 : u'ldap'" "

On another post where this question was asked, following options were given,
1. Install and configure the SA-ldapsearch component on your indexer instances.
2. Modify the commands.conf within SA-ldapsearch to run in the local space only.

However, I have already installed SA-ldapsearch on the Indexer but it didnt make any difference. And the drawback of 2nd solution is that it can result in degraded search performance due to restricting LDAP queries to the search head only.

Please advise how I can resolve this.

Thank you.

[hardikJsheth]

The error suggests it can't find ldap.conf file. Did you configure your AD/LDAP in SA-ldapsearch? You should have ldap.conf file within SA-ldapsearch either in default or in local folder like following:

[ADS]
alternatedomain = abc.bcd.com
basedn = dc=ads,dc=abc,dc=com
binddn = CN=SplunkUser,OU=Service Accounts,OU=User Accounts,DC=ads,DC=abc,DC=com
port = 3269
server = x.x.x.x
ssl = 1

[damode]

Hi @hardikJsheth,

There is already ldap.conf is the local dir of both S.H and Indexer. I had manually created ldap.conf on Indexer with same config as S.H, upon reading one of the posts on similar topic here.

Additional info : there is no passwords.conf and app.conf in the local folder of Indexer. S.H has all 3 files.

[hardikJsheth]

Since you have updated commands.conf file on your search head cluster to run query locally you don't need to configure app on your indexer.

In the configuration page it provides option to test your configuration. Did you try it out? Can you try this and if it fails look for the error in your splunkd.log file.

[damode]

Hi Hardik,

I didnt go for the 2nd option which is about editing the commands.conf just to avoid any impact on search performance. W.r.t architecture, I have just 1 Search head and 1 Indexer.

I have tried configuring through GUI and also got connection tested successfully. However, the issue with that page is, it shows connection untested when you either refresh or go back to that page. And whenever, I used to check the ldap.conf file, the ssl used to somehow get enabled, which I wanted to be disabled. Hence, I just do it manually.

[hardikJsheth]

The command is not working because you haven't configured ladp/AD account on indexer.

If you want to choose option 1 please configure ldap/AD credentials on indexer nodes. After the configuration you will have passwords.conf/ldap.conf on indexer.

[devd25]

Hi @Hardik,

Would copying the passwords.conf and app.conf of the Search head to the Indexer would work OR should I configure via the GUI on the indexer ?

[hardikJsheth]

You should configure it via GUI if you have Indexer's GUI available.

The passwords in passwords.conf are encrypted and for Splunk > =6.3, it ensures that same encryption keys are used between Search Nodes / indexer nodes which are in cluster. However I am not sure if Splunk manages the encryption keys between search head and indexers.

[damode]

I did it via GUI too, but still it gives that error.

Currently, both Search Head and the Indexer have the exact copy of app.conf, passwords.conf, ldap.conf

[damode]

This is what I found within the client.py at line 1653,

  try:
                response = self.get(key)
                return ConfigurationFile(self.service, PATH_CONF % key, state={'title': key})
            except HTTPError as he:
                if he.status == 404: # No entity matching key
                    raise KeyError(key) //**line 1653**
                else:
                    raise

How can I resolve this ?

I also noticed this folder (mentioned in the error message)
C:\Program Files\Splunk\var\run\searchpeers\SEARCH-HEAD-1511928556\apps\SA-ldapsearch has only bin folder. Should I create a local folder and place the ldap.conf and password.conf files here ?

[hardikJsheth]

Try it out. Copy past ladp.conf and password.conf to local folder.

[damode]

I had tried that but Splunk keeps updating/deleting these folders. The folder in which I had made those changes isn't there anymore.

Currently, its giving me the same error from C:\Program Files\Splunk\var\run\searchpeers\SEARCH-HEAD-1512451985\apps\SA-ldapsearch\bin\packages\splunklib\client.py

[hardikJsheth]

You can add following entries on your search head in distsearch.conf file under $SPLUNK_HOME/etc/system/local folder.

[replicationSettings:refineConf]
replicate.ldap               = true

This entry ensures that the ldap.conf is replicated to searchpeers folder.

[damode]

Hi Hardik,

The info on the distsearch.conf for the [replicationSettings:refineConf]
setting says that

These settings on their own do not cause files to be replicated. A file
must still be whitelisted (via
replicationWhitelist) to be eligible
for inclusion via these settings

So I am guessing I will have to add a file to the whitelist as well. is that right ? if yes, then please advise how I can do it via replicationWhitelist.

[hardikJsheth]

I have these settings

[replicationSettings:refineConf]
replicate.ldap               = true

[replicationWhitelist]
allConf = *.conf

You can also refer this splunk answers for ldapsearch error
https://answers.splunk.com/answers/312136/after-upgrading-to-splunk-63-why-is-the-splunk-sup.html

[damode]

Please ignore my previous comment regarding S.H not starting. Its resolved now.

At this stage, since ldap is already configured on Indexer, would the above settings override Indexer's current ldap settings ? or
should I delete the current configuration on Indexer and wait for Search head's ldap config to get replicated to the Indexer ?

I had already referred that link you shared, but it has to many varying answers with no concrete points. for e.g, the 1st answer deals with version prior to 2.1.2 but I am already running the latest version 2.1.4 still the issue persists.

Most of those talk about manually installing the ldap app on Indexer and pushing the config from the S.H via deployer. However with my architecture (1 S.H and 1 Indexer) how can I push the config to the Indexer so that it uses the same password hashes ? OR would your above provided settings achieve that task ?

[hardikJsheth]

No it won't impact the configurations under $SPLUNK_HOME/etc/apps. This changes impacts what gets into $SPLUNK_HOME/var/run/searchpeers folder.

[damode]

Hi Hardik,

I have got only one Search Head and one Indexer. Would the above settings have any negative impact on the S.H considering these [shclustering] settings are applicable for a search head cluster ?

[hardikJsheth]

I am sorry, the settings for shcluster are not needed. You need to add distsearch.conf as mentioned in the comments.