Splunk Support for Active Directory: Why does ldapsearch take 6-8 hours to return results?

[gwalford]

Splunk Support for Active Directory (SA-ldapsearch) is installed and configured on my search heads. When running the test on any system that has it installed, the test function completes.

When trying to run nearly any ldap search, it will take hours to return results.

For example, this search for the App Search Activity:

ldapsearch domain=`SA-LDAPSearch-Domain` search="(&(objectclass=user)(!(objectclass=computer)))"| fields description title cn dn sn name displayName givenName whenChanged sAMAccountName mail manager c l o st telephoneNumber department company directReports physicalDeliveryOfficeName | makemv directReports tokenizer="(?i)(CN=.*?dc=\S*)"| eval NumDirectReports = if(isnull(directReports),0,mvcount(directReports)) | fields - directReports _raw _time

Takes about 7 hours to return results.

We do have an insanely large domain here. I am sure that this is part of the reason.

Unfortunately, the Splunk App for Windows Infrastructure isn't populating a lot of drop-downs, and I suspect it is all related to the SA-ldapsearch search time returns.

[javiergn]

Hi, not sure if I might be able to help but I'm curious about this issue because I've never seen it before.

Three questions to start with:

• how many users would you expect your query to return?

• If you just run the ldapsearch bit without any extra processing, how long does it take roughly?

  ldapsearch domain=SA-LDAPSearch-Domain search="(&(objectclass=user)(!(objectclass=computer)))"

• If you run the following query from PowerShell, does it take that long too? I'm just trying to discard domain performance problems so please run it from the same server ldapsearch is running from

  Get-ADUser -Filter *

EDIT:

If your PowerShell command is much faster to complete then this is what I did on my previous client of mine and it worked like a charm. The end goal was slightly different as we just wanted a CSV to translate user SIDs into account names and therefore we used a lookup. In your case a csv should do the trick:

• Schedule PowerShell command to extract all your users into a CSV file every X hours
• Place CSV file inside the relevant directory within your app
• Use inputcsv to read from the file

I know this is not a solution for the ldap slow performance but it might be a good workaround.

I also did this with by exporting the users into an SQL db and then reading that via dblookup. Performance was similar but the SQL option allowed me to do funky stuff in SQL before bringing the data into Splunk.

Hope that helps.

Thanks,
J

[gwalford]

In Powershell "Get-ADUser -Filter *" begins returning results almost instantly - I don't know how long they would take to complete, but I can tell you the domain is insanely large - I don't even know what the sum total of the users would be.

The Splunk search
| ldapsearch domain="default" search="(&(objectclass=user)(!(objectclass=computer)))
Stalls with 0.0% of the time range scanned - this is where I need to extend the job to have it complete. My best guess is that the full user lookup takes hours to stream back from LDAP, but I don't really know (Will verify with my AD admins).

[ctaf]

I have the exact same problem. When I narrow down the search to 1 user (limit=1), it takes 7 seconds to complete the search. I am thinking the LDAP client is badly developped.

[javiergn]

Hi I have modified my answer above to include a workaround I used some time ago.
Hope that helps.

Thanks,
J