TL;DR - python script (ldapsearch) stops after 15 minutes. Didn't happen after upgrading the add-on. Happened after upgrading Splunk.
So, a few days before I upgraded Splunk from 6.3.1 to 6.5.1, I upgraded Splunk Support for Active Directory (SA-ldapsearch) add-on to 2.1.4. We have nightly jobs that search our directories for metadata and output to CSV. Historically, some of these searches run well over an hour. Upgrading to 2.1.4 didn't break these longer searches, but upgrading Splunk to 6.5.1 did. Here are some interesting points I have noticed:
After upgrading Splunk, dispatch.fetch seems to cap out right around 900 seconds. Normally, the search in question runs about 6,000-6,500 seconds.
If I search _audit for an affected search_id, at 15 minutes into the search, there is an event, but it's not from the user who owns the search, it's for splunk-system-user. Before upgrading to 6.5.1, this event does not occur.
Audit:[timestamp=01-06-2017 22:15:08.803, id=16356, user=splunk-system-user, action=search, info=granted REST: /search/jobs/scheduler_amg5NjMyOS1kcw_U0EtbGRhcHNlYXJjaA__RMD5382a042170949a29_at_1483758000_696][ctVE9bBqZ5wadW/RBmx70tVR3GbFX+my52Itx5qin3z9Lg0Kwn3fgFJoJBXGwiE3lKSDJyHa8VuFalijSW2MqDRCoNJOyA+gm1orBvAwKhUaLGS/s0eoQfPOwLThOMUJwmYyNQndkIE9l5M1rZPmjkxGtJLKW71Zdyb7FUGGU8Y=]
It's almost as if there is a new limit which was introduced by the upgrade, but I'm having trouble tracking down what limit this might be related to. As expected, there aren't any local or default limits.conf in the add-on. If I add filters to make the ldapsearch run faster (less than 15 minutes), the search works exactly like I would expect it to.
... View more