https://community.splunk.com/t5/Splunk-Search/Search-hangs-on-quot-TcpOutbound-quot-error/m-p/558432#M158632 <P>A scheduled search is hanging when it approaches around 28% completion. In search.log, the following message appears shortly before the search begins to experience issues</P><PRE>07-06-2021 07:21:30.846 WARN SearchResultCollator - Collector xxx-xxxx-xxxx produced chunk with startTime 1625255401.000000 when our cursor time was already 1625255234.000000, time ordering has failed! 07-06-2021 07:21:31.273 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW 07-06-2021 07:21:31.284 INFO SearchParser - PARSING: noop 07-06-2021 07:21:31.284 INFO DispatchExecutor - BEGIN OPEN: Processor=noop 07-06-2021 07:21:31.284 INFO SearchEvaluator - Searched for keyword in results without _raw field 07-06-2021 07:21:31.284 INFO DispatchExecutor - END OPEN: Processor=noop 07-06-2021 07:21:31.284 INFO PreviewExecutor - Finished preview generation in 0.000866621 seconds. 07-06-2021 07:21:32.373 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW 07-06-2021 07:21:43.188 INFO TcpOutbound - Received unexpected socket close condition with unprocessed data in RX buffer. Processing remaining bytes=11014 of data in RX buffer. socket_status="Connection closed by peer" paused=1</PRE><P>Appears to be related to&nbsp;<A title="https://community.splunk.com/t5/Deployment-Architecture/Splunk-search-on-SH-return-Unknown-error-for-indexers/m-p/511827" href="https://community.splunk.com/t5/Deployment-Architecture/Splunk-search-on-SH-return-Unknown-error-for-indexers/m-p/511827" target="_self">this</A>&nbsp;issue, but I don't see any additional log entries mentioning the ulimit value.&nbsp;</P> Tue, 06 Jul 2021 14:56:18 GMT mh393 2021-07-06T14:56:18Z https://community.splunk.com/t5/Splunk-Search/Search-hangs-on-quot-TcpOutbound-quot-error/m-p/558432#M158632 <P>A scheduled search is hanging when it approaches around 28% completion. In search.log, the following message appears shortly before the search begins to experience issues</P><PRE>07-06-2021 07:21:30.846 WARN SearchResultCollator - Collector xxx-xxxx-xxxx produced chunk with startTime 1625255401.000000 when our cursor time was already 1625255234.000000, time ordering has failed! 07-06-2021 07:21:31.273 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW 07-06-2021 07:21:31.284 INFO SearchParser - PARSING: noop 07-06-2021 07:21:31.284 INFO DispatchExecutor - BEGIN OPEN: Processor=noop 07-06-2021 07:21:31.284 INFO SearchEvaluator - Searched for keyword in results without _raw field 07-06-2021 07:21:31.284 INFO DispatchExecutor - END OPEN: Processor=noop 07-06-2021 07:21:31.284 INFO PreviewExecutor - Finished preview generation in 0.000866621 seconds. 07-06-2021 07:21:32.373 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW 07-06-2021 07:21:43.188 INFO TcpOutbound - Received unexpected socket close condition with unprocessed data in RX buffer. Processing remaining bytes=11014 of data in RX buffer. socket_status="Connection closed by peer" paused=1</PRE><P>Appears to be related to&nbsp;<A title="https://community.splunk.com/t5/Deployment-Architecture/Splunk-search-on-SH-return-Unknown-error-for-indexers/m-p/511827" href="https://community.splunk.com/t5/Deployment-Architecture/Splunk-search-on-SH-return-Unknown-error-for-indexers/m-p/511827" target="_self">this</A>&nbsp;issue, but I don't see any additional log entries mentioning the ulimit value.&nbsp;</P> Tue, 06 Jul 2021 14:56:18 GMT https://community.splunk.com/t5/Splunk-Search/Search-hangs-on-quot-TcpOutbound-quot-error/m-p/558432#M158632 mh393 2021-07-06T14:56:18Z