It seems to be taking much longer for dispatched search request to finalize. I see in search.log a lot of POST requests from Timeliner. Is Timeliner new in 6.5.0?
10-25-2016 23:01:47.210 INFO Timeliner - Sending POST request 'https://10.0.0.88:8089/services/search/remote/splunk02/1477436470.3/events?offset=231976&count=689' 10-25-2016 23:01:47.415 INFO Timeliner - https://10.0.0.88:8089 No work units available, moving to idle 10-25-2016 23:01:47.452 INFO Timeliner - Stats[10.0.0.88] 16 / 117.00 / 297.00 / 2879.00 / 558025.00 / 49.99 / 850457 10-25-2016 23:01:47.452 INFO Timeliner - Stats[10.0.0.99] 16 / 58.00 / 198.00 / 2179.00 / 362025.00 / 63.87 / 384405 10-25-2016 23:01:47.452 INFO Timeliner - Stats[Collections] 16 / 275.00 / 2872.00 / 24850.00 / 49518646.00 / 826.27 / 76817158 10-25-2016 23:01:47.459 INFO Timeliner - Stats[10.0.0.88] 1 / 93.00 / 0.00 / 93.00 / 8649.00 / 0.00 / 554549 10-25-2016 23:01:47.459 INFO Timeliner - Stats[Collections] 2 / 0.00 / 111.00 / 111.00 / 12321.00 / 55.50 / 291866418
Sorry my response is a little late but I have just recently come across the same issue. As I understand if remote_timeline is enabled (default option) in limits.conf then timeline will be built remotely. This is not a new behavior with 6.5.0.
I have seen that increasing "maxchunkqueue_size" in the limits.conf file can improve search improvement. It has by default increased from 1MB to 10MB between versions 6.3.x and 6.5.x. You may like to increase this further.
Also, the following parameters can also effect this behavior:
fetchremotesearchlog = disabledSavedSearches
remotetimeline = true
remotetimelinefetchall = 1
Setting them as follows has been seen to have a positive impact on the search speed:
fetchremotesearchlog = false
remotetimeline = true
remotetimelinefetchall = false
However, there may be other reasons for the searches not finalizing but without reviewing the full log files it is difficult to say.
I was also wondering if you manage to resolve the issue and what steps you took?
Any comment on what "No work units available, moving to idle" actually means?
What functionality would I lose if I applied your changes?
To clarify, you see those are "INFO" events, right? Not "ERROR" or "WARN". Just want to make sure you don't go after a red herring.