All Apps and Add-ons

Splunk Support for Active Directory: After upgrading to Splunk 6.5.1, why does scheduled search stop after 900 seconds?

Path Finder

TL;DR - python script (ldapsearch) stops after 15 minutes. Didn't happen after upgrading the add-on. Happened after upgrading Splunk.

So, a few days before I upgraded Splunk from 6.3.1 to 6.5.1, I upgraded Splunk Support for Active Directory (SA-ldapsearch) add-on to 2.1.4. We have nightly jobs that search our directories for metadata and output to CSV. Historically, some of these searches run well over an hour. Upgrading to 2.1.4 didn't break these longer searches, but upgrading Splunk to 6.5.1 did. Here are some interesting points I have noticed:

  1. After upgrading Splunk, dispatch.fetch seems to cap out right around 900 seconds. Normally, the search in question runs about 6,000-6,500 seconds.
  2. If I search _audit for an affected search_id, at 15 minutes into the search, there is an event, but it's not from the user who owns the search, it's for splunk-system-user. Before upgrading to 6.5.1, this event does not occur.

    Audit:[timestamp=01-06-2017 22:15:08.803, id=16356, user=splunk-system-user, action=search, info=granted REST: /search/jobs/scheduleramg5NjMyOS1kcwU0EtbGRhcHNlYXJjaA_RMD5382a042170949a29at1483758000696][ctVE9bBqZ5wadW/RBmx70tVR3GbFX+my52Itx5qin3z9Lg0Kwn3fgFJoJBXGwiE3lKSDJyHa8VuFalijSW2MqDRCoNJOyA+gm1orBvAwKhUaLGS/s0eoQfPOwLThOMUJwmYyNQndkIE9l5M1rZPmjkxGtJLKW71Zdyb7FUGGU8Y=]

It's almost as if there is a new limit which was introduced by the upgrade, but I'm having trouble tracking down what limit this might be related to. As expected, there aren't any local or default limits.conf in the add-on. If I add filters to make the ldapsearch run faster (less than 15 minutes), the search works exactly like I would expect it to.

Explorer

Hi, I've found the solution to my problem. I hope it will also solve yours. Solution is to set in the [search] stanza of limits.conf (in etc/system or etc/apps/) the batchwaitafter_end parameter to a value higher than the longest duration of your ldapsearch queries. I had a look to all limit values and it was the only one that was equal to 900 (in seconds, e.g. 15 minutes).

Explorer

I have the same problem with 6.5.1 and have a search head and a indexer. I have noticed that if we don't get any hit whitin 15 min the search just stops and search.log reports no data found, however if any data is retrieved within 15 min the search can continue for hours with no problem.
I have not been able to break it down as some search can contine without hits, anyway increasing the batchwaitafter_end at the search head has solved it.

0 Karma

Path Finder

Bueller? Anybody?

0 Karma

Explorer

Hi, I have the same behaviour: search is stopped after 900 seconds (with 6.5.1 on Windows 2008 R2 and SA-ldapsearch v2.1.0). I opened case #402460 on 05/10/2016... which I accept to be closed on 29/11 because Splunk support did not find the root cause. Even given that the problem is systematic in our standalone Splunk environment.
Let's hope Splunk will take a deeper look now 😉
Emmanuel.

0 Karma

Explorer

You gave me the courage to open a new case: #441733. I'll let you know the outcome. Or perhaps a user will answer you before ;-).

0 Karma

Path Finder

I have a case open as well (#440445). So far, no luck with finding the root cause.

0 Karma

Splunk Employee
Splunk Employee

Hi @brettwilliams - Were you able to test out ecathalo's solution below? If yes and it worked, please don't forget to resolve this post by clicking on "Accept". If you still need more help, please provide a comment with some feedback. Thanks!

0 Karma

Path Finder

Forgot to mention something relevant from search.log which happens at the same time as item 2. Looks like this:
01-06-2017 22:00:05.381 INFO script - Writing search results info to /opt/splunk/var/run/splunk/dispatch...
01-06-2017 22:15:05.400 WARN DispatchThread - queryfinished or lower level infrastructure is notifying that query is done but the et is not yet set to Zero.

0 Karma