Splunk search throwing Job Terminated Unexpectedly

[poddraj]

Hi Team,

I have a simple search with index=test which is returning 2587 events with Timeframe of Week to Date. Same search when ran in Smart mode it is not returning the complete result and kept running for more than 2-3 mins and returning Job Terminated Unexpectedly error.
Can you please help where is the issue and how can I fix it. As I am unable to attach files am pasting the search.log file for my search.
Please advise as I am frequently getting this error.

01-09-2020 14:16:58.374 INFO dispatchRunner - Search process mode: preforked (first search in process) (build 1357bef0a7f6).
01-09-2020 14:16:58.374 INFO dispatchRunner - initing LicenseMgr in search process: nonPro=0
01-09-2020 14:16:58.374 INFO LicenseMgr - Initing LicenseMgr
01-09-2020 14:16:58.374 INFO LMConfig - serverName=delfdda3aws guid=18197C53-69CE-4DDE-A67C-2338604EF18E
01-09-2020 14:16:58.378 INFO LMConfig - connection_timeout=30
01-09-2020 14:16:58.378 INFO LMConfig - send_timeout=30
01-09-2020 14:16:58.378 INFO LMConfig - receive_timeout=30
01-09-2020 14:16:58.378 INFO LMConfig - key=license_warnings_update_interval not found in licenser stanza of server.conf, defaulting=0
01-09-2020 14:16:58.378 INFO LMConfig - squash_threshold=2000
01-09-2020 14:16:58.378 INFO LMConfig - strict_pool_quota=1
01-09-2020 14:16:58.378 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-09-2020 14:16:58.378 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
01-09-2020 14:16:58.378 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
01-09-2020 14:16:58.378 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=false
01-09-2020 14:16:58.378 INFO LMStackMgr - closing stack mgr
01-09-2020 14:16:58.378 INFO LMSlaveInfo - all slaves cleared
01-09-2020 14:16:58.378 INFO LMStackMgr - Initalized license_warnings_update_interval=auto
01-09-2020 14:16:58.381 INFO LMStack - Added type=download-trial license, from file=enttrial.lic, to stack=download-trial of group=Trial
01-09-2020 14:16:58.381 INFO LMStackMgr - created stack='download-trial'
01-09-2020 14:16:58.381 INFO LMStackMgr - have to auto-set active stack group='Trial' reason='invalid/missing group id' gidStr='' oldGid=Invalid
01-09-2020 14:16:58.381 INFO LMStackMgr - added pool auto_generated_pool_download-trial to stack download-trial
01-09-2020 14:16:58.381 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-09-2020 14:16:58.381 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
01-09-2020 14:16:58.381 INFO LMStackMgr - Initialized hideQuotaWarning = "0"
01-09-2020 14:16:58.381 INFO LMStackMgr - init completed [18197C53-69CE-4DDE-A67C-2338604EF18E,Trial,runContext_splunkd=false]
01-09-2020 14:16:58.381 INFO LicenseMgr - StackMgr init complete...
01-09-2020 14:16:58.381 INFO LMTracker - Setting default product type='enterprise'
01-09-2020 14:16:58.381 INFO LMTracker - this is not splunkd, will perform partial init
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SubgroupId state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-09-2020 14:16:58.381 INFO LicenseMgr - Tracker init complete...
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'licenses'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'pools'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'stacks'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'groups'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'slaves'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'localslave'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'licensermessages'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'scriptedwarning'
01-09-2020 14:16:58.406 INFO AdminManagerDispatch - added factory for admin handler: 'licenseusage'
01-09-2020 14:16:58.406 INFO dispatchRunner - registering build time modules, count=1
01-09-2020 14:16:58.406 INFO dispatchRunner - registering search time components of build time module name=vix
01-09-2020 14:16:58.406 INFO dispatchRunner - Getting search configuration data from: /apps/opt/delphi/Bala/splunk/etc/modules/parsing/config.xml
01-09-2020 14:16:58.409 INFO BundlesSetup - Setup stats for /apps/opt/delphi/Bala/splunk/etc: wallclock_elapsed_msec=23, cpu_time_used=0.009999, shared_services_generation=2, shared_services_population=1
01-09-2020 14:16:58.432 INFO UserManagerPro - Load authentication: forcing roles="admin, power, user"
01-09-2020 14:16:58.432 INFO PipelineComponent - Ingestion pipeline sets have been configured to use pipeline set policy=round_robin for multiple pipeline sets.
01-09-2020 14:16:58.469 INFO UserManager - Setting user context: splunk-system-user
01-09-2020 14:16:58.469 INFO UserManager - Done setting user context: NULL -> splunk-system-user
01-09-2020 14:16:58.470 INFO UserManager - Unwound user context: splunk-system-user -> NULL
01-09-2020 14:16:58.470 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.470 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.470 INFO dispatchRunner - search context: user="admin", app="search", bs-pathname="/apps/opt/delphi/Bala/splunk/etc"
01-09-2020 14:16:58.477 INFO SearchParser - PARSING: search index=main
01-09-2020 14:16:58.477 INFO dispatchRunner - SearchHeadInitSearchMs=8
01-09-2020 14:16:58.477 INFO SearchParser - PARSING: search index=main
01-09-2020 14:16:58.477 INFO SearchParser - PARSING: search index=main
01-09-2020 14:16:58.477 INFO dispatchRunner - Executing the Search orchestrator and iterator model (dfs=0).
01-09-2020 14:16:58.478 INFO SearchOrchestrator - SearchOrchestrator getting constructed
01-09-2020 14:16:58.478 INFO SearchOrchestrator - Initialized the SRI
01-09-2020 14:16:58.486 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
01-09-2020 14:16:58.488 INFO SearchOrchestrator - Initialzing the run time settings for the orchestrator.
01-09-2020 14:16:58.488 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.488 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.488 INFO SearchOrchestrator - Creating the search DAG.
01-09-2020 14:16:58.488 INFO SearchParser - PARSING: search index=main
01-09-2020 14:16:58.528 INFO SearchProcessor - Building search filter
01-09-2020 14:16:58.592 INFO SearchEvaluatorBasedExpander - Performing lookup expansions
01-09-2020 14:16:58.592 INFO SearchEvaluatorBasedExpander - Lookup expansion took 0 ms
01-09-2020 14:16:58.592 INFO SearchEvaluatorBasedExpander - Performing calculated field expansions
01-09-2020 14:16:58.592 INFO SearchEvaluatorBasedExpander - Calculated field expansion took 0 ms
01-09-2020 14:16:58.592 INFO SearchEvaluatorBasedExpander - Performing field alias expansions
01-09-2020 14:16:58.593 INFO SearchEvaluatorBasedExpander - field alias expansions took 0 ms
01-09-2020 14:16:58.593 INFO SearchEvaluatorBasedExpander - Performing kvfield expansions
01-09-2020 14:16:58.593 INFO SearchEvaluatorBasedExpander - kvfield expansions took 0 ms
01-09-2020 14:16:58.593 INFO SearchEvaluatorBasedExpander - Performing reverse sourcetype expansions
01-09-2020 14:16:58.593 INFO SearchEvaluatorBasedExpander - sourcetype expansions took 0 ms
01-09-2020 14:16:58.593 INFO UnifiedSearch - Expanded index search = index=main
01-09-2020 14:16:58.593 INFO UnifiedSearch - base lispy: [ AND index::main ]
01-09-2020 14:16:58.593 INFO UnifiedSearch - Processed search targeting arguments
01-09-2020 14:16:58.593 INFO DispatchThread - BatchMode: allowBatchMode: 0, conf(1): 1, timeline/Status buckets(0):0, realtime(0):0, report pipe empty(0):1, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
01-09-2020 14:16:58.594 INFO DispatchThread - Setup timeliner partialCommits=1
01-09-2020 14:16:58.594 INFO DispatchThread - required fields list to add to remote search = _bkt,_cd,_si,host,index,linecount,source,sourcetype,splunk_server
01-09-2020 14:16:58.594 INFO SearchParser - PARSING: fields keepcolorder=t "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.594 INFO DispatchCommandProcessor - summaryHash=32c0d4f22a96501c summaryId=18197C53-69CE-4DDE-A67C-2338604EF18E_search_admin_32c0d4f22a96501c remoteSearch=litsearch index=main | fields keepcolorder=t "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.594 INFO DispatchCommandProcessor - summaryHash=NSd2ffc33f9b6ec5ae summaryId=18197C53-69CE-4DDE-A67C-2338604EF18E_search_admin_NSd2ffc33f9b6ec5ae remoteSearch=litsearch index=main | fields keepcolorder=t "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.594 INFO DispatchThread - Getting summary ID for summaryHash=NSd2ffc33f9b6ec5ae
01-09-2020 14:16:58.610 INFO SearchParser - PARSING: search index=main
01-09-2020 14:16:58.610 INFO UnifiedSearch - Processed search targeting arguments
01-09-2020 14:16:58.610 INFO AstOptimizer - SrchOptMetrics optimize_toJson=0.000211702
01-09-2020 14:16:58.611 INFO AstVisitorFactory - Not building visitor : replace_datamodel_stats_cmds_with_tstats
01-09-2020 14:16:58.611 INFO SearchParser - PARSING: | search index=main
01-09-2020 14:16:58.612 INFO ProjElim - Black listed processors=[addinfo]
01-09-2020 14:16:58.612 INFO AstOptimizer - Search optimizations have been disabled in limits.conf. Set enabled=true in [search_optimization::replace_stats_cmds_with_tstats]
01-09-2020 14:16:58.612 INFO AstVisitorFactory - Not building visitor : replace_stats_cmds_with_tstats
01-09-2020 14:16:58.612 INFO AstOptimizer - SrchOptMetrics optimization=0.001138538
01-09-2020 14:16:58.612 INFO SearchPhaseGenerator - Optimized Search =| search index=main
01-09-2020 14:16:58.612 INFO ScopedTimer - search.optimize 0.001588635
01-09-2020 14:16:58.612 INFO ParallelReducePolicy - Current Search Head doesn't have any usable peers to use.
01-09-2020 14:16:58.612 INFO PhaseNodeGenerationVisitor - User lacking run_multi_phased_searches, rolling back to 2-phase mode.
01-09-2020 14:16:58.612 INFO PhaseToPipelineVisitor - Phase Search = | search index=main
01-09-2020 14:16:58.612 INFO SearchParser - PARSING: | search index=main
01-09-2020 14:16:58.622 INFO SearchProcessor - Building search filter
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Performing lookup expansions
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Lookup expansion took 0 ms
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Performing calculated field expansions
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Calculated field expansion took 0 ms
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Performing field alias expansions
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - field alias expansions took 0 ms
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Performing kvfield expansions
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - kvfield expansions took 0 ms
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - Performing reverse sourcetype expansions
01-09-2020 14:16:58.647 INFO SearchEvaluatorBasedExpander - sourcetype expansions took 0 ms
01-09-2020 14:16:58.648 INFO UnifiedSearch - Expanded index search = index=main
01-09-2020 14:16:58.648 INFO UnifiedSearch - base lispy: [ AND index::main ]
01-09-2020 14:16:58.648 INFO UnifiedSearch - Processed search targeting arguments
01-09-2020 14:16:58.648 INFO PhaseToPipelineVisitor - Phase Search =
01-09-2020 14:16:58.648 INFO SearchPipeline - ReportSearch=0 AllowBatchMode=0
01-09-2020 14:16:58.648 INFO SearchPhaseParserControl - Adding SimpleResultsCombiner to merge remote input into correct time order.
01-09-2020 14:16:58.648 INFO SearchParser - PARSING: simpleresultcombiner max=0
01-09-2020 14:16:58.648 INFO SearchPhaseGenerator - Storing only 1000 events per timeline buckets due to limits.conf max_events_per_bucket setting.
01-09-2020 14:16:58.648 INFO SearchPhaseGenerator - No need for RTWindowProcessor
01-09-2020 14:16:58.648 INFO SearchPhaseGenerator - Adding timeliner to final phase
01-09-2020 14:16:58.648 INFO SearchParser - PARSING: | timeliner remote=0 partial_commits=1 max_events_per_bucket=1000 fieldstats_update_maxperiod=60 bucket=300 extra_field=*
01-09-2020 14:16:58.660 INFO TimelineCreator - Creating timeline with remote=0 partialCommits=1 commitFreq=0 syncKSFreq=0 maxSyncKSPeriodTime=60000 bucket=300 latestTime=1578579418.000000 earliestTime=1577836800.000000
01-09-2020 14:16:58.660 INFO SearchPhaseGenerator - required fields list to add to different pipelines = ,_bkt,_cd,_si,host,index,linecount,source,sourcetype,splunk_server
01-09-2020 14:16:58.660 INFO SearchPhaseGenerator - Remote Timeliner=
01-09-2020 14:16:58.660 INFO SearchPhaseGenerator - Fileds=fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.660 INFO SearchPhaseGenerator - REMOTE TIMELINER ADDED
01-09-2020 14:16:58.661 INFO SearchParser - PARSING: fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.661 INFO SearchPhaseGenerator - Search Phases created.
01-09-2020 14:16:58.662 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.662 INFO UserManager - Done setting user context: admin -> admin
01-09-2020 14:16:58.662 INFO UserManager - Unwound user context: admin -> admin
01-09-2020 14:16:58.662 INFO DistributedSearchResultCollectionManager - Stream search: litsearch index=main | fields keepcolorder=t "" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.663 INFO ExternalResultProvider - No external result providers are configured
01-09-2020 14:16:58.663 INFO DistributedSearchResultCollectionManager - ERP_FACTORY initialized, but zero external result provider, hence disabling _isERPCollectionEnabled
01-09-2020 14:16:58.663 INFO DistributedSearchResultCollectionManager - Default search group:*
01-09-2020 14:16:58.663 INFO SearchTargeter - Index list is not available for delfdda3aws targeting based on splunk_server and splunk_server_group only
01-09-2020 14:16:58.663 INFO DistributedSearchResultCollectionManager - Connecting to peer delfdda3aws connectAll 0 connectToSpecificPeer 1
01-09-2020 14:16:58.663 INFO SearchPhaseGenerator - Time spends on creating distributed search results infrastructure; dispatchcreatedSearchResultInfrastructure=0.001032853 seconds.
01-09-2020 14:16:58.663 INFO SearchParser - PARSING: | streamnoop
01-09-2020 14:16:58.663 INFO SearchParser - PARSING: streamnoop | timeliner remote=0 partial_commits=1 max_events_per_bucket=1000 fieldstats_update_maxperiod=60 bucket=300 extra_field=*
01-09-2020 14:16:58.664 INFO TimelineCreator - Creating timeline with remote=0 partialCommits=1 commitFreq=0 syncKSFreq=0 maxSyncKSPeriodTime=60000 bucket=300 latestTime=1578579418.000000 earliestTime=1577836800.000000
01-09-2020 14:16:58.664 INFO SearchOrchestrator - Starting the status control thread.
01-09-2020 14:16:58.664 INFO SearchOrchestrator - Starting phase=1
01-09-2020 14:16:58.670 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.671 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.671 INFO SearchStatusEnforcer - Enforcing disk quota = 10485760000
01-09-2020 14:16:58.671 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.671 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.673 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.673 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.684 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.684 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.684 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.684 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.684 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.684 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.684 INFO ReducePhaseExecutor - Stating phase_1
01-09-2020 14:16:58.684 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.684 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.695 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.695 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.695 INFO SearchParser - PARSING: litsearch index=main | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server"
01-09-2020 14:16:58.711 INFO SearchParser - PARSING: typer | tags
01-09-2020 14:16:58.730 INFO FastTyper - found nodes count: comparisons=6, unique_comparisons=5, terms=4, unique_terms=4, phrases=12, unique_phrases=12, total leaves=22
01-09-2020 14:16:58.738 INFO PreviewExecutor - Preview Enforcing initialization done
01-09-2020 14:16:58.747 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.747 INFO UserManager - Done setting user context: admin -> admin
01-09-2020 14:16:58.747 INFO FastSearchFilter - Finished initializing IndexScopedFilter - trivial=1, nTerms=0, oTerms=0, host=0, source=0, sourcetype=0, linecount=0 exactCustomCmp=0
01-09-2020 14:16:58.747 INFO UserManager - Unwound user context: admin -> admin
01-09-2020 14:16:58.747 INFO CMBucketId - CMIndexId: New indexName=main inserted, mapping to id=1
01-09-2020 14:16:58.748 INFO IndexScopedSearch - ct=1578579418.000000 et=1577836800.000000 lt=1578579418.000000 dbsize=14
01-09-2020 14:16:58.748 INFO UnifiedSearch - Initialization of search data structures took 38 ms
01-09-2020 14:16:58.748 INFO UnifiedSearch - Processed search targeting arguments
01-09-2020 14:16:58.755 INFO LocalCollector - Final required fields list = *,Message,_bkt,_cd,_raw,_si,_subsecond,host,index,linecount,source,sourcetype,splunk_server
01-09-2020 14:16:58.755 INFO UserManager - Unwound user context: admin -> NULL
01-09-2020 14:16:58.755 INFO UserManager - Setting user context: admin
01-09-2020 14:16:58.755 INFO UserManager - Done setting user context: NULL -> admin
01-09-2020 14:16:58.887 WARN SearchOperator:kv - buildRegexList provided empty conf key, ignoring.
01-09-2020 14:16:58.906 INFO TimelineCreator - Commit timeline at cursor=1578576120.000000
01-09-2020 14:16:59.459 INFO Timeliner - cannot clear local results, still have 8 results. Ratio of results written to disk vs total is: 1 wrote n to disk: 82
01-09-2020 14:16:59.460 INFO TimelineCreator - Commit timeline at cursor=1578471435.000000
01-09-2020 14:16:59.747 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:17:09.465 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:17:09.467 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:17:19.473 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:17:19.475 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:17:29.131 WARN SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=0 currentSetStart=1578471435.000000
01-09-2020 14:17:29.131 INFO ResultsCollationProcessor - just a keepalive, ignoring
01-09-2020 14:17:39.207 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:17:39.210 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:17:49.546 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:17:49.548 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:17:59.131 WARN SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=1 currentSetStart=1578471435.000000
01-09-2020 14:17:59.131 INFO ResultsCollationProcessor - just a keepalive, ignoring
01-09-2020 14:18:09.133 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:18:09.135 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:18:19.136 INFO TimelineCreator - Commit timeline at cursor=9223372036854775807.999999
01-09-2020 14:18:19.138 INFO ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW
01-09-2020 14:18:29.132 WARN SearchResultWorkUnit - timed out, sending keepalive nConsecutiveKeepalive=2 currentSetStart=1578471435.000000
01-09-2020 14:18:29.132 INFO ResultsCollationProcessor - just a keepalive, ignoring
01-09-2020 14:18:37.079 FATAL ProcessRunner - Unexpected EOF from process runner child!
01-09-2020 14:18:37.146 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!