Slow performance Splunk 6.0.1 Master Head with Splunk 5.0.2 Indexer

[lpolo]

The same query was run in two master heads. These are the results:

Response time from master head Splunk 5.0.2:

This search has completed and has returned 12 results by scanning 41,283 events in 11.072seconds. ->
9.604 seconds dispatch.stream.remote

Response time from master head splunk 6.0.1:

This search has completed and has returned 12 results by scanning 41,283 events in208.412 seconds.
200.562 dispatch.finalizeRemoteTimeline

The same behavior across all splunk 6.0.1 master head but not in splunk 5.0.2 master head.

Is there anyone in the forum with the same problem?

Any idea?

Lp

[lpolo]

this is the solution to this problem:

Make the following changes in
/$splunkHome$/etc/system/local/limits.conf

[search]
remote_timeline_fetchall = 0
fetch_remote_search_log = false

then restart splunkd

[rabitoblanco]

Trying to get a handle on what "generate a remote timeline" means-- what's the effect of disabling this?

[lpolo]

All Indexers were upgraded to Splunk 6.0.1. however, the response time from the master head 6.0.1 continues to be slower compared to splunk 5.0.2 master head.
For the sake of clarity, both master heads are running with the same type of hardware.
Open case with splunk tech support.
any idea?

[lpolo]

I found this:

Remote timeline feature can reduce performance
A 6.x search head by default asks its search peers to generate a remote timeline. This isn't a problem with 6.x search peers, but 5.x search peers won't know how to generate the timeline. As a result, searches can slow dramatically.
The workaround is to add the following attribute to limits.conf on the search head :
[search]
remote_timeline_fetchall = false
After making this change, you must restart the search head.
Important: You should remove this attribute after all search peers have been upgraded to 6.x.