ReducePhaseExecutor taking too much time to load dashboard- How to make more efficient

[jeyakumar8]

Hi Team,

I'm using the sim command to stream data from signalFx and created a dashboard using "Monitoring infrastructure add-on"

For certain searches with high resolution value, search is stuck with 06-21-2021 06:05:21.656 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW

After the max wait time search got auto cancelled.

resolution value used = 7 days in milliseconds

Successful job search log

06-21-2021 04:30:02.885 INFO  UserManager - Done setting user context: NULL -> admin
06-21-2021 04:30:02.884 INFO  UserManager - Done setting user context: NULL -> admin
06-21-2021 04:30:02.885 INFO  ReducePhaseExecutor - Stating phase_1
06-21-2021 04:30:02.885 INFO  SearchStatusEnforcer - Enforcing disk quota = 10485760000
06-21-2021 04:30:02.885 INFO  DispatchExecutor - BEGIN OPEN: Processor=stats
06-21-2021 04:30:02.925 INFO  PreviewExecutor - Preview Enforcing initialization done
06-21-2021 04:30:03.199 INFO  DispatchExecutor - END OPEN: Processor=stats
06-21-2021 04:30:03.199 INFO  DispatchExecutor - BEGIN OPEN: Processor=sort
06-21-2021 04:30:03.200 INFO  DispatchExecutor - END OPEN: Processor=sort
06-21-2021 04:30:03.201 INFO  DispatchExecutor - BEGIN OPEN: Processor=table
06-21-2021 04:30:03.201 INFO  DispatchExecutor - END OPEN: Processor=table
06-21-2021 04:30:03.201 INFO  DispatchExecutor - BEGIN OPEN: Processor=noop
06-21-2021 04:30:03.201 INFO  DispatchExecutor - END OPEN: Processor=noop
06-21-2021 04:30:03.207 INFO  ReducePhaseExecutor - Ending phase_1
06-21-2021 04:30:03.207 INFO  UserManager - Unwound user context: admin -> NULL

Failed job log

06-21-2021 06:05:20.756 INFO  UserManager - Setting user context: admin
06-21-2021 06:05:20.756 INFO  UserManager - Done setting user context: NULL -> admin
06-21-2021 06:05:20.756 INFO  UserManager - Setting user context: admin
06-21-2021 06:05:20.756 INFO  SearchStatusEnforcer - Enforcing disk quota = 10485760000
06-21-2021 06:05:20.756 INFO  UserManager - Done setting user context: NULL -> admin
06-21-2021 06:05:20.756 INFO  ReducePhaseExecutor - Stating phase_1
06-21-2021 06:05:20.756 INFO  DispatchExecutor - BEGIN OPEN: Processor=stats
06-21-2021 06:05:20.756 INFO  PreviewExecutor - Preview Enforcing initialization done
06-21-2021 06:05:21.656 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
06-21-2021 06:18:12.557 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL
06-21-2021 06:18:12.557 INFO  DispatchExecutor - User applied action=CANCEL while status=0
06-21-2021 06:18:12.557 ERROR SearchStatusEnforcer - sid:admin__admin_c2ltLXJlcG9ydHMtdWk__search4_1624280720.3608 Search auto-canceled
06-21-2021 06:18:12.557 INFO  SearchStatusEnforcer - State changed to FAILED due to: Search auto-canceled

Please help on this.

[Kenji]

I had the exact same issue. ReducePhaseExecutor got stuck and the search was canceled. In my case, the root cause was an event contains a special character "U+00a0".