Job terminated unexpectedly with no obvious error if too many rex mode=sed commands in query

[john_arrowwood]

I am trying to summarize AWS ELB Access Logs.  Once I get the raw URLs, I need to substitute out the path and query parameters so that like paths collapse into the same string.  I do this via a series of rex commands.  For example:

| rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:gif|jpg|jpeg|png|ico)(?= |\?|;)/\1{image}/"
| rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:css)(?= |\?|;)/\1{stylesheet}/"
| rex field=pattern mode=sed "s/(.*?\/)[^ \/]+?\.(?i)(?:js)(?= |\?|;)/\1{script}/"
| rex field=pattern mode=sed "s/(\/(?:bootstrap|jquery|timeout))\/[0-9]+(?:\.[0-9])*(?=\/)/\1\/{version}/"

Everything works fine until I reach a certain number of replacements, and then the job fails.  I delete enough, and suddenly it starts working again.  I look in the search job inspector, and don't see anything obvious.  Same for the Job Details Dashboard, and the search.log.  I see nothing to indicate what the problem was.  Here's the tail end of the search.log:

07-13-2023 11:37:03.695 INFO  SearchParser [29042 localCollectorThread] - PARSING: typer | tags
07-13-2023 11:37:03.900 INFO  FastTyper [29042 localCollectorThread] - found nodes count: comparisons=2208, unique_comparisons=1080, terms=187, unique_terms=116, phrases=156, unique_phrases=112, total leaves=2551
07-13-2023 11:37:03.917 INFO  BatchSearch [29042 localCollectorThread] - Using Batch Search
07-13-2023 11:37:03.918 INFO  UserManager [29042 localCollectorThread] - Setting user context: k28500
07-13-2023 11:37:03.918 INFO  UserManager [29042 localCollectorThread] - Done setting user context: k28500 -> k28500
07-13-2023 11:37:03.919 INFO  FastSearchFilter [29042 localCollectorThread] - Finished initializing IndexScopedFilter - trivial=0, nTerms=0, oTerms=0, host=0, source=1, sourcetype=1, linecount=0 exactCustomCmp=0
07-13-2023 11:37:03.919 INFO  UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> k28500
07-13-2023 11:37:03.919 INFO  BatchSearch [29042 localCollectorThread] -  BatchSearch is initialized for indexes = {aws-data}, et = 1689261722.000000000, lt = 1689262622.000000000, index_et = -9223372036854775808.000000000, index_lt = 9223372036854775807.999999000, noRead = FALSE
07-13-2023 11:37:03.919 INFO  BatchSearch [29042 localCollectorThread] - Clearing any DDM references
07-13-2023 11:37:03.919 INFO  BatchSearch [29042 localCollectorThread] - index: aws-data dbsize=0
07-13-2023 11:37:03.919 INFO  UnifiedSearch [29042 localCollectorThread] - Initialization of search data structures took 225 ms
07-13-2023 11:37:03.920 INFO  UnifiedSearch [29042 localCollectorThread] - Processed search targeting arguments
07-13-2023 11:37:03.921 INFO  StatsProcessorV2 [29042 localCollectorThread] - StatsV2 (stats) args: count, by, pattern
07-13-2023 11:37:03.921 INFO  StatsProcessorV2 [29042 localCollectorThread] - StatsProcessorV2::processArguments: Unaligned accesses are free
07-13-2023 11:37:03.921 INFO  StatsAggregations [29042 localCollectorThread] - Instantiating Stats function group_count for key=, alias=count
07-13-2023 11:37:03.921 INFO  StatsContext [29042 localCollectorThread] - shouldFallbackToOldStats: _use_v2_level=USE_V2_ALL, fallback=false
07-13-2023 11:37:03.921 INFO  AutoLookupDriver [29042 localCollectorThread] - Will use Lookup: LOOKUP-Domain-host
07-13-2023 11:37:03.921 INFO  AutoLookupDriver [29042 localCollectorThread] - Will use Lookup: LOOKUP-cf-domains-ags-env
07-13-2023 11:37:03.938 WARN  SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-EMR
07-13-2023 11:37:03.942 WARN  SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-extraction_1_kp
07-13-2023 11:37:03.942 WARN  SearchOperator:kv [29042 localCollectorThread] - Could not find a transform named REPORT-extracting_fi_pattern_name
07-13-2023 11:37:03.949 INFO  LocalCollector [29042 localCollectorThread] - Final required fields list = AGS,AppAbbrev,ClientRequestHost,EventData_Xml,LogFileCleared_Xml,Message,RenderingInfo_Xml,System_Props_Xml,UserData_Xml,_raw,_subsecond,affectedResources,eventName,extended_properties,functionName,host,index,prestats_reserved_*,psrsvd_*,raw_msg,request,request_query,request_uri,source,sourcetype
07-13-2023 11:37:03.949 INFO  UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:03.949 INFO  UserManager [29042 localCollectorThread] - Setting user context: k28500
07-13-2023 11:37:03.949 INFO  UserManager [29042 localCollectorThread] - Done setting user context: NULL -> k28500
07-13-2023 11:37:03.949 INFO  UserManager [29042 localCollectorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.432 INFO  ReducePhaseExecutor [29032 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
07-13-2023 11:37:04.489 INFO  SearchParser [29032 StatusEnforcerThread] - PARSING: | streamnoop
07-13-2023 11:37:04.489 INFO  SearchParser [29032 StatusEnforcerThread] - PARSING: streamnoop  | stats  count by pattern
07-13-2023 11:37:04.489 INFO  StatsContext [29032 StatusEnforcerThread] -  Setting chunk size min=65536 max=1048576 double_every=100
07-13-2023 11:37:04.489 INFO  StatsContext [29032 StatusEnforcerThread] -  Setting max memory usage to 209715200
07-13-2023 11:37:04.489 INFO  StatsProcessorV2 [29032 StatusEnforcerThread] - StatsV2 (stats) args: count, by, pattern
07-13-2023 11:37:04.489 INFO  StatsProcessorV2 [29032 StatusEnforcerThread] - StatsProcessorV2::processArguments: Unaligned accesses are free
07-13-2023 11:37:04.489 INFO  StatsAggregations [29032 StatusEnforcerThread] - Instantiating Stats function group_count for key=, alias=count
07-13-2023 11:37:04.489 INFO  StatsContext [29032 StatusEnforcerThread] - shouldFallbackToOldStats: _use_v2_level=USE_V2_ALL, fallback=false
07-13-2023 11:37:04.489 INFO  ReducePhaseExecutor [29032 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW
07-13-2023 11:37:04.490 INFO  CMBucketId [29040 SearchResultExecutorThread] - CMIndexId: New indexName=aws-data inserted, mapping to id=2
07-13-2023 11:37:04.492 INFO  PreviewExecutor [29032 StatusEnforcerThread] - Finished preview generation in 0.000485272 seconds.
07-13-2023 11:37:04.826 INFO  UserManager [29036 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO  UserManager [29038 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO  ResultsCollationProcessor [29043 phase_1] - Writing remote_event_providers.csv to disk
07-13-2023 11:37:04.826 INFO  UserManager [29040 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO  UserManager [29037 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.826 INFO  UserManager [29039 SearchResultExecutorThread] - Unwound user context: k28500 -> NULL
07-13-2023 11:37:04.829 INFO  DownloadRemoteDataTransaction [29043 phase_1] - Downloading logs from all remote event providers
07-13-2023 11:37:05.027 INFO  ReducePhaseExecutor [29043 phase_1] - Downloading all remote search.log files took 0.199 seconds
07-13-2023 11:37:05.027 INFO  DownloadRemoteDataTransaction [29043 phase_1] - Downloading logs from all remote event providers
07-13-2023 11:37:05.188 INFO  ReducePhaseExecutor [29043 phase_1] - Downloading all remote search_telemetry.json files took 0.162 seconds

Any advice? 

[VatsalJagani]

@john_arrowwood - It seems like resource issue for your Splunk environment.

• How long the search runs before terminating?
• How much memory/CPU do you have in your environment?
• Do your Splunk admins apply any resource restriction in your Splunk environment, in terms of time limit, memory limit, storage limit, max no. of searches limit, etc.?

I hope this helps!!!

[john_arrowwood · Accepted solution]

I run the query over the last 15 minutes.  Or a very specific time window.  There is a threshold.  The raw data being read is the same.  But if I add one too many `rex` commands, it dies.

But it appears to die at the very end, after already generating the results.  

I don't know about CPU/Memory, I'm just a user, not an admin.

I did post the question to StackOverflow, too, and got a recommendation to use `| eval field=replace(field, pattern, replacement)` instead of `| rex mode=sed` - I'm trying that now to see if it solves the problem.

[john_arrowwood]

YES.  Replacing `| rex mode=sed` with `| eval field=replace(field, {regex}, {replace)` solved my problem.

[VatsalJagani]

@john_arrowwood - Kindly accept your comment so future community members can get help from it. You can do that by clicking on the "Accept as Solution" button.

Splunk Community Moderator,

Vatsal