https://docs.splunk.com/Documentation/Splunk/9.2.1/ReleaseNotes/Fixedissues https://docs.splunk.com/Documentation/Splunk/9.1.4/ReleaseNotes/Fixedissues One  customer reported a very interesting issue with graceful splunk restart. Event missing during a graceful restart/rolling restart(splunk stop gracefully finished). useACK=true is an option but that ideally must be applied if splunk stop timed-out. This has been an issue for so many years. This is important where config changes are pushed frequently, thus triggering frequent indexer/HF/IF restart. The issue is fixed by 9.1.4/9.2.1   TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown   How to detect if it's applicable for your deployment? Check splunkd.log for  WARN TcpInputProc - Could not process data received from network. Aborting due to shutdown Also from metrics.log see https://community.splunk.com/t5/Knowledge-Management/During-indexer-restart-indexer-cluster-rolling-restart/m-p/683763#M9962 ... View more

https://docs.splunk.com/Documentation/SplunkCloud/9.1.2312/Data/Usepersistentqueues Persistent queuing is available for certain types of inputs, but not all. One major limitation with persistent queue is that if it's enabled on certain UF/HF/IHF/IUF inputs, downstream parsingqueue/indexqueue/tcpoutqueue are blocked/saturated and a DS bundle push triggers splunk restart, events will be dropped if UF/HF/IHF/IUF failed to drain queues. On windows DC, persistent queuing is enabled for windows modular inputs, DS bundle push triggers DC restart and events in parsingqueue/tcpoutqueue will be dropped. On windows DC, windows event logs are lost when Laptops are off VPN and shutdown, in-memory queue events are dropped.  When Laptops are off the network and restarted/shutdown, in-memory queue events are dropped.  However if persistent queue is available at tcpoutqueue, it will prevent any kind of loss when splunk restart happens regardless of input type of the event.  Splunk 9.4 ( future release) has added persistent queue at tcpout. Eliminating the need to enable persistent queue on certain supported inputs. Hopefully it will solve persistent queue need arising due to unavoidable  restarts/ rolling restarts/off network laptops. Benefits: It's at no additional cost. On windows DC, winevent logs are not lost when Laptops are off VPN and are shutdown. Output PQ will write in-memory events on to disk during splunk shutdown. Forwarders can use splunk useACK( which can be used only between splunk to splunk instances).  No loss of metadata information. No degraded splunk indexing performance. Provide persistent queue support for modinputs running on forwarder. Provide persistent queue support for HEC server running on forwarder. Writes all in-memory/in-flight events to output persistent queue during splunk restart or a back-pressure. Instead of paid PQ route  With third party PQ, during splunk restart on forwarding tier, still splunk in-memory queued events might get dropped.  ... View more

index=_internal source=*splunkd.log* host=<all indexer hosts> bucketreplicator full earliest=-15m | stats count dc(host) as num_indexer_blocked_by_peer by peer | where num_indexer_blocked_by_peer > 0 AND count > 0 | join type=left peer [ search index=_introspection host=<all indexer hosts> hostwide earliest=-10m | stats values(data.instance_guid) as peer by host] ... View more

9.1.3/9.2.1 onwards slow indexer/receiver detection capability is fully functional now (SPL-248188, SPL-248140).   https://docs.splunk.com/Documentation/Splunk/9.2.1/ReleaseNotes/Fixedissues You can enable it on forwarding side in outputs.conf maxSendQSize = <integer> * The size of the tcpout client send buffer, in bytes. If tcpout client(indexer/receiver connection) send buffer is full, a new indexer is randomly selected from the list of indexers provided in the server setting of the target group stanza. * This setting allows forwarder to switch to new indexer/receiver if current indexer/receiver is slow. * A non-zero value means that max send buffer size is set. * 0 means no limit on max send buffer size. * Default: 0 Additionally 9.1.3/9.2.1 and above will correctly log target ipaddress causing tcpout blocking.   WARN AutoLoadBalancedConnectionStrategy [xxxx TcpOutEloop] - Current dest host connection nn.nn.nn.nnn:9997, oneTimeClient=0, _events.size()=20, _refCount=2, _waitingAckQ.size()=4, _supportsACK=1, _lastHBRecvTime=Thu Jan 20 11:07:43 2024 is using 20214400 bytes. Total tcpout queue size is 26214400. Warningcount=20   Note: This config works correctly starting 9.1.3/9.2.1. Do not use it with 9.2.0/9.1.0/9.1.1/9.1.2( there is incorrect calculation https://community.splunk.com/t5/Getting-Data-In/Current-dest-host-connection-is-using-18446603427033668018-bytes/m-p/678842#M113450). ... View more

During graceful indexer/HF restart/stop (basically where ever splunktcp is configured) if you see last entries in metrics.log before splunk finally stops.  Where splunktcpin queue (name=splunktcpin) shows current_size, largest_size, smallest_size has same value( but parsingqueue to indexqueue none blocked), TcpInputProcessor fails to drain splunktcpin queue despite parsingqueue or indexqueue are empty.    02-18-2024 00:54:28.370 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1507, largest_size=1507, smallest_size=1507 02-18-2024 00:54:28.370 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=indexqueue, max_size_kb=10240, current_size_kb=7, current_size=40, largest_size=40, smallest_size=0 02-18-2024 00:54:28.368 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1148, largest_size=1148, smallest_size=1148 02-18-2024 00:54:28.368 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=indexqueue, max_size_kb=10240, current_size_kb=7, current_size=40, largest_size=40, smallest_size=0 02-18-2024 00:53:57.364 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1507, largest_size=1507, smallest_size=1507 02-18-2024 00:53:57.364 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:53:57.362 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1148, largest_size=1148, smallest_size=1148 02-18-2024 00:53:57.362 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:53:26.372 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1507, largest_size=1507, smallest_size=1507 02-18-2024 00:53:26.372 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:53:26.370 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1148, largest_size=1148, smallest_size=1148 02-18-2024 00:53:26.370 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:52:55.371 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1507, largest_size=1507, smallest_size=0 02-18-2024 00:52:55.371 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:52:55.369 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=splunktcpin, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1148, largest_size=1148, smallest_size=0 02-18-2024 00:52:55.369 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:52:24.397 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=splunktcpin, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=30, smallest_size=0 02-18-2024 00:52:24.396 +0000 INFO Metrics - group=queue, ingest_pipe=1, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 02-18-2024 00:52:24.380 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=splunktcpin, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=16, smallest_size=0 02-18-2024 00:52:24.380 +0000 INFO Metrics - group=queue, ingest_pipe=0, name=indexqueue, max_size_kb=10240, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0   During graceful shutdown pipeline processors are expected to drain the queue. This issue is fixed in 9.2.1 and 9.1.4.  ... View more

There is a practice of setting queueSize in inputs.conf [http://<token>] stanza. queueSize over writes server.conf stanza     [queue=httpInputQ] maxSize   Now if you have multiple tokens with different queueSize.     inputs.conf [http://1] queueSize=1 [http://2] queueSize=2 [http://3] queueSize=3 [http://4] queueSize=4     Globally only one inputs.conf stanza wins for final httpInputQ size. This setting should only be set if setting 'persistentQueueSize' as well. If there are multiple http inputs configured and each input has set 'queueSize' but persistentQueueSize is not is set, splunkd will create one in-memory queue and pick the 'queueSize' value from first stanza after sorting http stanzas with matching token of first received http event in ascending order. With multiple pipelines configured, each pipeline will create one in-memory queue depending on the first http event received by the pipeline thus each pipeline might have different sized httpInputQ created. If there are multiple http stanzas configured and 'persistentQueueSize' is not set, prefer to set 'maxSize' under 'queue=httpInputQ' stanza in server.conf. So best practice would be to never set per token queueSize in inputs.conf. Instead set one time in server.conf, if not setting persistentQueueSize.     [queue=httpInputQ] maxSize     ... View more