Splunk Search

Can someone explain what a SearchResultWorkUnit is and why it would timeout?

grittonc
Contributor

I am using the SDK to create my first custom search command. I'm using the Splunk Free version to test it out.

It works great for relatively small numbers of records (10-50). For larger record counts (100+) about 20 records are processed before I see this in search.log:

06-02-2020 15:53:18.427 WARN  SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=0 currentSetStart=1591052862.000000
06-02-2020 15:53:18.427 INFO  ResultsCollationProcessor - just a keepalive, ignoring
06-02-2020 15:53:28.428 INFO  TimelineCreator - Commit timeline at cursor=2147483647.000000
06-02-2020 15:53:28.429 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
06-02-2020 15:53:38.430 INFO  TimelineCreator - Commit timeline at cursor=2147483647.000000
06-02-2020 15:53:38.430 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW

This block repeats until my maxwait setting is reached. Then I see this block.

06-02-2020 15:57:48.524 WARN  NetUtils - select_for timeout hit waiting for read
06-02-2020 15:57:48.524 WARN  NetUtils - readWithTimeout failed. ret=-2
06-02-2020 15:57:48.524 ERROR ChunkedExternProcessor - Error or timeout while attempting to read transport header (Resource temporarily unavailable)
06-02-2020 15:57:48.679 ERROR ChunkedExternProcessor - Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.679 ERROR LocalCollector - sid: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.679 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 ERROR DispatchThread - sid:1591127547.502 Search results might be incomplete: the search process on the local peer:CL2 ended prematurely. Check the local peer log, such as $SPLUNK_HOME/var/log/splunk/splunkd.log and as well as the search.log for the particular search.
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  ReducePhaseExecutor - Ending phase_1
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.680 ERROR SearchOrchestrator - Phase_1 failed due to : Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.680 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=CANCEL
06-02-2020 15:57:48.680 INFO  DispatchExecutor - User applied action=CANCEL while status=0
06-02-2020 15:57:48.680 ERROR SearchStatusEnforcer - sid:1591127547.502 Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.680 INFO  SearchStatusEnforcer - State changed to FAILED due to: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.
06-02-2020 15:57:48.684 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.684 INFO  DispatchManager - DispatchManager::dispatchHasFinished(id='1591127547.502', username='admin')
06-02-2020 15:57:48.684 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.685 INFO  ISearchOperator - 0x7f348fb51d00 PREAD_HISTOGRAM: usec_1_8=21 usec_8_64=0 usec_64_512=0 usec_512_4096=0 usec_4096_32768=0 usec_32768_262144=0 usec_262144_INF=0 
06-02-2020 15:57:48.687 INFO  UserManager - Unwound user context: NULL -> NULL
06-02-2020 15:57:48.687 INFO  LookupProviderFactory - Clearing out lookup shared provider map
06-02-2020 15:57:48.689 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'urlstatus' command: Invalid message received from external search command during search, see search.log.

Can someone explain what a SearchResultWorkUnit is and why it would timeout? And why would the keepalive be ignored? Is there a setting I can change somewhere?

Labels (1)
0 Karma

andrew_burnett
Path Finder

I keep seeing something similiar, except mine doesn't have any error message.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...