Here are the configs for on-prem customers willing to apply and avoid adding more hardware cost. 9.4.0 and above most of the indexing configs are automated that’s why dropped from 9.4.0 suggested list. Note: Assuming replication queue is full for most of the indexers and as a result indexing pipeline is also full however indexers do have plenty of idle cpu and IO is not an issue. On-prem Splunk version 9.4.0 and above Indexes.conf [default] maxMemMB=100 Server.conf [general] autoAdjustQueue=true ( It can be applied on on any splunk instance UF/HF/SH/IDX) Splunk version 9.1 to 9.3.x Indexes.conf [default] maxMemMB=100 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0 Server.conf [general] parallelIngestionPipelines = 4 [queue=indexQueue] maxSize=500MB [queue=parsingQueue] maxSize=500MB [queue=httpInputQ] maxSize = 500MB maxMemMB, will try to minimize creation of tsidx files as much as possible at the cost of higher memory usage by mothership(main splunkd). maxConcurrentOptimizes, on indexing side it’s internally 1 no matter what the setting is set to. But on target replication side launching more splunk-optimize processes means pausing receiver until that splunk-optimize process is launched. So reducing it to keep receiver do more of indexing work than launching splunk-optimize process. With 9.4.0, both source (indexprocessor) and target(replication in thread) will internally auto adjust it to 1. maxRunningProcessGroups, allow more splunk-optimize processes concurrently. With 9.4.0, it's auto. processTrackerServiceInterval, run splunk-optimize processes ASAP. With 9.4.0, you don't have to change. parallelIngestionPipelines, have more receivers on target side. With 9.4.0, you can enable auto scaling of  pipelines. maxSize, don’t let huge batch ingestion by HEC client block queues and receive 503. With 9.4.0 autoAdjustQueue set to true, it's no more a fix size. ... View more

Here are the configs for on-prem customers willing to apply and avoid adding more hardware cost. 9.4.0 and above most of the indexing configs are automated that’s why dropped from 9.4.0 suggested list. Note: Assuming replication queue is full for most of the indexers and as a result indexing pipeline is also full however indexers do have plenty of idle cpu and IO is not an issue. On-prem Splunk version 9.4.0 and above Indexes.conf [default] maxMemMB=100 Server.conf [general] autoAdjustQueue=true Splunk version 9.1 to 9.3.x Indexes.conf [default] maxMemMB=100 maxConcurrentOptimizes=2 maxRunningProcessGroups=32 processTrackerServiceInterval=0 Server.conf [general] parallelIngestionPipelines = 4 [queue=indexQueue] maxSize=500MB [queue=parsingQueue] maxSize=500MB [queue=httpInputQ] maxSize = 500MB maxMemMB, will try to minimize creation of tsidx files as much as possible at the cost of higher memory usage by mothership(main splunkd). maxConcurrentOptimizes, on indexing side it’s internally 1 no matter what the setting is set to. But on target replication side launching more splunk-optimize processes means pausing receiver until that splunk-optimize process is launched. So reducing it to keep receiver do more of indexing work than launching splunk-optimize process. With 9.4.0, both source (indexprocessor) and target(replication in thread) will internally auto adjust it to 1. maxRunningProcessGroups, allow more splunk-optimize processes concurrently. With 9.4.0, it's auto. processTrackerServiceInterval, run splunk-optimize processes ASAP. With 9.4.0, you don't have to change. parallelIngestionPipelines, have more receivers on target side. With 9.4.0, you can enable auto scaling of  pipelines. maxSize, don’t let huge batch ingestion by HEC client block queues and receive 503. With 9.4.0 autoAdjustQueue set to true, it's no more a fix size queue. ... View more

Are you recommending enableOldS2SProtocol=true? Are you implementing  enableOldS2SProtocol=true? If yes, read below. Splunk has dropped support for oldest S2S version. However added enableOldS2SProtocol config to allow forwarder use oldest protocol. With enableOldS2SProtocol=true, forwarder is allowed to  use oldest protocol (protocol level 0). First ever protocol. You are essentially using almost 20 years old protocol. With enableOldS2SProtocol=false, forwarder is allowed to  use minimum protocol level 1 with negotiateProtocolLevel config. If negotiateProtocolLevel is not set( by default not set), then forwarder and receiver will be negotiating latest common protocol supported by forwarder and receiver. If you are on Splunk 9.2.x receiver and forwarder is 9.0.x and above, then protocol 6 is being used. When protocol negotiation happens between fwd and receiver, if the receiver says  protocol 0, fwd does not accept that and still use minimum supported protocol 1 unless enableOldS2SProtocol=true is set on fwd. Suggesting enableOldS2SProtocol=true on fwd means receiver is only capable of protocol 0 and forcing fwd to use protocol 0. Suggesting enableOldS2SProtocol=true and negotiateProtocolLevel=0 on fwd means fwd is forced to use protocol 0 regardless of receiver's protocol level. Protocol levels. 0: Maximum network traffic over S2S connection. 1: Network traffic optimization over S2S connection. 2: Additional network traffic optimization over S2S connection. 3: Metric support. 4: Ack support for rawless metric events. 5: Flag potential dup events. 6: Flag for cloned metric events so that cloned events exempted from license usage. 7: SSL certificate requests Make an informed decision. ... View more

Apart from https://community.splunk.com/t5/Splunk-Enterprise/Linear-memory-growth-with-Splunk-9-4-0-and-above/m-p/712550#M21712 Some splunk instances(UF/HF/SH/IDX) might see higher memory usage after the upgrade. 9.4.0 has introduced new active channel cache. It has a cache TTL of 3600 sec. active_eligibility_age = <integer> * The time, in seconds, after which splunkd removes an idle input channel from the active channel cache to free up memory. * Default: 3600 Before 9.4.0, splunkd was using inactive channel cache. It had a cache ttl of 330 sec. It's not used anymore. inactive_eligibility_age_seconds = <integer> * Time, in seconds, after which an inactive input channel will be removed from the cache to free up memory. * Default: 330 Because of high active channel cache TTL, splunkd memory footprint might be higher on some splunk deployments. In limits.conf reduce active channel cache TTL to 330 ( 9.4.2 onwards, by default it's 330)     [input_channels] active_eligibility_age = 330         ... View more