I am trying to summarize AWS ELB Access Logs. Once I get the raw URLs, I need to substitute out the path and query parameters so that like paths collapse into the same string. I do this via a series of rex commands. For example: | rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:gif|jpg|jpeg|png|ico)(?= |\?|;)/\1{image}/" | rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:css)(?= |\?|;)/\1{stylesheet}/" | rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:js)(?= |\?|;)/\1{script}/" | rex field=pattern mode=sed "s/(\/(?:bootstrap|jquery|timeout))\/[0-9]+(?:\.[0-9])*(?=\/)/\1\/{version}/" Everything works fine until I reach a certain number of replacements, and then the job fails. I delete enough, and suddenly it starts working again. I look in the search job inspector, and don't see anything obvious. Same for the Job Details Dashboard, and the search.log. I see nothing to indicate what the problem was. Here's the tail end of the search.log: 07-13-2023 11:37:03.695 INFO SearchParser [29042 localCollectorThread] - PARSING: typer | tags
07-13-2023 11:37:03.900 INFO FastTyper [29042 localCollectorThread] - found nodes count: comparisons=2208, unique_comparisons=1080, terms=187, unique_terms=116, phrases=156, unique_phrases=112, total leaves=2551
07-13-2023 11:37:03.917 INFO BatchSearch [29042 localCollectorThread] - Using Batch Search
07-13-2023 11:37:03.918 INFO UserManager [29042 localCollectorThread] - Setting user context: k28500
07-13-2023 11:37:03.918 INFO UserManager [29042 localCollectorThread] - Done setting user context: k28500 -> k28500
07-13-2023 11:37:03.919 INFO FastSearchFilter [29042 localCollectorThread] - Finished initializing IndexScopedFilter - trivial=0, nTerms=0, oTerms=0, host=0, source=1, sourcetype=1, linecount=0 exactCustomCmp=0
07-13-2023 11:37:03.919 INFO UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> k28500
07-13-2023 11:37:03.919 INFO BatchSearch [29042 localCollectorThread] - BatchSearch is initialized for indexes = {aws-data}, et = 1689261722.000000000, lt = 1689262622.000000000, index_et = -9223372036854775808.000000000, index_lt = 9223372036854775807.999999000, noRead = FALSE
07-13-2023 11:37:03.919 INFO BatchSearch [29042 localCollectorThread] - Clearing any DDM references
07-13-2023 11:37:03.919 INFO BatchSearch [29042 localCollectorThread] - index: aws-data dbsize=0
07-13-2023 11:37:03.919 INFO UnifiedSearch [29042 localCollectorThread] - Initialization of search data structures took 225 ms
07-13-2023 11:37:03.920 INFO UnifiedSearch [29042 localCollectorThread] - Processed search targeting arguments
07-13-2023 11:37:03.921 INFO StatsProcessorV2 [29042 localCollectorThread] - StatsV2 (stats) args: count, by, pattern
07-13-2023 11:37:03.921 INFO StatsProcessorV2 [29042 localCollectorThread] - StatsProcessorV2::processArguments: Unaligned accesses are free
07-13-2023 11:37:03.921 INFO StatsAggregations [29042 localCollectorThread] - Instantiating Stats function group_count for key=, alias=count
07-13-2023 11:37:03.921 INFO StatsContext [29042 localCollectorThread] - shouldFallbackToOldStats: _use_v2_level=USE_V2_ALL, fallback=false
07-13-2023 11:37:03.921 INFO AutoLookupDriver [29042 localCollectorThread] - Will use Lookup: LOOKUP-Domain-host
07-13-2023 11:37:03.921 INFO AutoLookupDriver [29042 localCollectorThread] - Will use Lookup: LOOKUP-cf-domains-ags-env
07-13-2023 11:37:03.938 WARN SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-EMR
07-13-2023 11:37:03.942 WARN SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-extraction_1_kp
07-13-2023 11:37:03.942 WARN SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-extracting_fi_pattern_name
07-13-2023 11:37:03.949 INFO LocalCollector [29042 localCollectorThread] - Final required fields list = AGS,AppAbbrev,ClientRequestHost,EventData_Xml,LogFileCleared_Xml,Message,RenderingInfo_Xml,System_Props_Xml,UserData_Xml,_raw,_subsecond,affectedResources,eventName,extended_properties,functionName,host,index,prestats_reserved_*,psrsvd_*,raw_msg,request,request_query,request_uri,source,sourcetype
07-13-2023 11:37:03.949 INFO UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:03.949 INFO UserManager [29042 localCollectorThread] - Setting user context: k28500
07-13-2023 11:37:03.949 INFO UserManager [29042 localCollectorThread] - Done setting user context: NULL -> k28500
07-13-2023 11:37:03.949 INFO UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.432 INFO ReducePhaseExecutor [29032 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
07-13-2023 11:37:04.489 INFO SearchParser [29032 StatusEnforcerThread] - PARSING: | streamnoop
07-13-2023 11:37:04.489 INFO SearchParser [29032 StatusEnforcerThread] - PARSING: streamnoop | stats count by pattern
07-13-2023 11:37:04.489 INFO StatsContext [29032 StatusEnforcerThread] - Setting chunk size min=65536 max=1048576 double_every=100
07-13-2023 11:37:04.489 INFO StatsContext [29032 StatusEnforcerThread] - Setting max memory usage to 209715200
07-13-2023 11:37:04.489 INFO StatsProcessorV2 [29032 StatusEnforcerThread] - StatsV2 (stats) args: count, by, pattern
07-13-2023 11:37:04.489 INFO StatsProcessorV2 [29032 StatusEnforcerThread] - StatsProcessorV2::processArguments: Unaligned accesses are free
07-13-2023 11:37:04.489 INFO StatsAggregations [29032 StatusEnforcerThread] - Instantiating Stats function group_count for key=, alias=count
07-13-2023 11:37:04.489 INFO StatsContext [29032 StatusEnforcerThread] - shouldFallbackToOldStats: _use_v2_level=USE_V2_ALL, fallback=false
07-13-2023 11:37:04.489 INFO ReducePhaseExecutor [29032 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
07-13-2023 11:37:04.490 INFO CMBucketId [29040 SearchResultExecutorThread] - CMIndexId: New indexName=aws-data inserted, mapping to id=2
07-13-2023 11:37:04.492 INFO PreviewExecutor [29032 StatusEnforcerThread] - Finished preview generation in 0.000485272 seconds.
07-13-2023 11:37:04.826 INFO UserManager [29036 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO UserManager [29038 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO ResultsCollationProcessor [29043 phase_1] - Writing remote_event_providers.csv to disk
07-13-2023 11:37:04.826 INFO UserManager [29040 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO UserManager [29037 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO UserManager [29039 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.829 INFO DownloadRemoteDataTransaction [29043 phase_1] - Downloading logs from all remote event providers
07-13-2023 11:37:05.027 INFO ReducePhaseExecutor [29043 phase_1] - Downloading all remote search.log files took 0.199 seconds
07-13-2023 11:37:05.027 INFO DownloadRemoteDataTransaction [29043 phase_1] - Downloading logs from all remote event providers
07-13-2023 11:37:05.188 INFO ReducePhaseExecutor [29043 phase_1] - Downloading all remote search_telemetry.json files took 0.162 seconds Any advice?
... View more