Can someone explain what a SearchResultWorkUnit is and why it would timeout?

[grittonc]

I am using the SDK to create my first custom search command. I'm using the Splunk Free version to test it out.

It works great for relatively small numbers of records (10-50). For larger record counts (100+) about 20 records are processed before I see this in search.log:

06-02-2020 15:53:18.427 WARN  SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=0 currentSetStart=1591052862.000000
06-02-2020 15:53:18.427 INFO  ResultsCollationProcessor - just a keepalive, ignoring
06-02-2020 15:53:28.428 INFO  TimelineCreator - Commit timeline at cursor=2147483647.000000
06-02-2020 15:53:28.429 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
06-02-2020 15:53:38.430 INFO  TimelineCreator - Commit timeline at cursor=2147483647.000000
06-02-2020 15:53:38.430 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW

This block repeats until my maxwait setting is reached. Then I see this block.

06-02-2020 15:57:48.524 WARN  NetUtils - select_for timeout hit waiting for read
06-02-2020 15:57:48.524 WARN  NetUtils - readWithTimeout failed. ret=-2
06-02-2020 15:57:48.524 ERROR ChunkedExternProcessor - Error or timeout while attempting to read transport header (Resource temporarily unavailable)
06-02-2020 15:57:48.679 ERROR ChunkedExternProcessor - Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.679 ERROR LocalCollector - sid: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.679 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 ERROR DispatchThread - sid:1591127547.502 Search results might be incomplete: the search process on the local peer:CL2 ended prematurely. Check the local peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  ReducePhaseExecutor - Ending phase_1
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 ERROR SearchOrchestrator - Phase_1 failed due to : Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.680 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL
06-02-2020 15:57:48.680 INFO  DispatchExecutor - User applied action=CANCEL while status=0
06-02-2020 15:57:48.680 ERROR SearchStatusEnforcer - sid:1591127547.502 Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.680 INFO  SearchStatusEnforcer - State changed to FAILED due to: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.684 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.684 INFO  DispatchManager - DispatchManager::dispatchHasFinished(id='1591127547.502', username='admin')
06-02-2020 15:57:48.684 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.685 INFO  ISearchOperator - 0x7f348fb51d00 PREAD_HISTOGRAM: usec_1_8=21 usec_8_64=0 usec_64_512=0 usec_512_4096=0 usec_4096_32768=0 usec_32768_262144=0 usec_262144_INF=0 
06-02-2020 15:57:48.687 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.687 INFO  LookupProviderFactory - Clearing out lookup shared provider map
06-02-2020 15:57:48.689 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.

Can someone explain what a SearchResultWorkUnit is and why it would timeout? And why would the keepalive be ignored? Is there a setting I can change somewhere?

[andrew_burnett]

I keep seeing something similiar, except mine doesn't have any error message.