Could I clarify some points here please? @hrawat wrote: Forwarder applies backoff setting (default 300 sec) based on following default settings ####### # Backoff Settings When Unable To Send Events to Indexer # The settings in this section determine forwarding behavior when there are # repeated failures in sending events to an indexer ("sending failures"). ####### secsInFailureInterval = <integer> * The number of seconds contained in a failure interval. * If the number of write failures to the indexer exceeds 'maxFailuresPerInterval' in the specified 'secsInFailureInterval' seconds, the forwarder applies backoff. * The backoff time period range is 1-10 * 'autoLBFrequency'. * Default: 1 backoffOnFailure = <positive integer> * The number of seconds a forwarder backs off, or stops sending events, before attempting to make another connection with the indexer. * Default: 30 Under the section "secsInFailureInterval", there is a mention of "* The backoff time period range is 1-10 * 'autoLBFrequency'." Under the backoffOnFailure , it advises the default backoff time is 30 seconds. And you have written "Forwarder applies backoff setting (default 300 sec) ". How does the time period add up to 300 seconds?   I do also note that the setting: autoLBFrequencyIntervalOnGroupFailure Needs to be lower than the autoLBFrequency, so I'm assuming the autoLBFrequency must be set to a reasonable value to use this setting? I normally set the autoLBFrequency quite low when using asynchronous load balancing.   Thanks ... View more

While this is about the indexing tier, I had one clarification in relation to  autoAdjustQueue = true In server.conf in Splunk versions above 9.4.0 on heavy forwarders. Does this setting relate to outputs.conf queue sizing as well?   I was thinking that if I tune my outputs.conf queue sizes and I use the related  pipelineSetAutoScale = true Then I've sized the outputs.conf queues but I'm unsure how many pipelines will be running if it scales up the pipeline count...which would normally change the size of queue I would use in outputs.conf ... View more

To clarify my understanding, I believe the recommendation is to set in outputs.conf: compressed = true useClientSSLCompression = true As the compressed=true will do no harm on legacy forwarders (I've checked the docs back until 7.0 and this parameter appears to be quite old), and the useClientSSLCompression will still work as expected *until* the indexing or intermediate forwarding tier moves to TLS1.3 in a future Splunk version...at which point the compressed=true will be used automatically by the forwarders (including legacy versions). I believe there will be no issues with having both set to true on legacy forwarders with the current Splunk releases that still support TLS1.2... Is that correct? ... View more

A quick clarification on the 9.4.0 settings for server.conf, you have mentioned On-prem Splunk version 9.4.0 and above Indexes.conf [default] maxMemMB=100 Server.conf [general] autoAdjustQueue=true The spec file for server.conf appears to show that autoAdjustQueue under the [queue] stanza, should it be under [queue]? With the indexes.conf setting, does that number multiply out based on the number of indexes configured? Should I be more cautious when having 1000 indexes configured vs having 100 indexes configured? I'm unsure when the "max memory" usage might occur from that setting... Thanks ... View more

I noticed that https://docs.splunk.com/Documentation/Splunk/9.4.0/Admin/Serverconf does not mention prometheus. Is this an undocumented feature that is getting disabled to prevent a memory leak issue? ... View more

In the past I've used outputgroup = <string> on the inputs.conf of [http] stanzas It sounds like the versions mentioned (and newer versions) now support: _TCP_ROUTING _meta   And a few other settings, is that correct? It is nice to have the product match it's spec file documentation 🙂   Thanks ... View more

Persistent queue support for monitor inputs will be very useful once it's available. ... View more

Thanks for the information, I assume the target is to fix this in a future UF 9.3.x release? Furthermore, would you happen to know what would happen if the setting was accidentally applied on a HF?   Clients of our deployment server will sometimes run a Splunk enterprise version instead of a UF so I suspect we will need to be careful... ... View more

It may be worth adding that the acknowledgement option cannot protect against data loss a scenario where a forwarder is restarted while the remote endpoint is not available To expand on this point, let's assume we have universal forwarder A, sending data to heavy forwarder B (and only HF B). (And then assume B connects to indexers) If A is reading from a file and sending to B, if we shutdown B, and while B is unable to process data we restart A during this downtime of B, any "in memory" data is lost at this point as the memory buffer if flushed on shutdown. The file monitor will re-read the portion of the file *after* the lost portion of the data.   This experiment is quite easy to setup in a development environment, the only point I'm adding is that (as advertised) the acknowledgement protects against intermediate data loss. It does not protect against data loss when the remote endpoint is down and the source is restarted. ... View more

Exactly. Disabling default apps is a bit tricky. Combine this with the 9.1.x version running a search on every instance and then an automated method like this helps on large indexer clusters, cluster managers et cetera. The application works on most apps, I did find the splunk assist app cannot be disabled using the REST API. Thanks @isoutamo ... View more

Yes that's the actual WARN message, the worst I've seen is a warning count of 9001 with a 150MB queue, the forwarder itself forwards a peak of over 100MB/s 05-21-2024 18:48:47.099 +1000 WARN AutoLoadBalancedConnectionStrategy [264180 TcpOutEloop] - Current dest host connection 10.x.x.x:9997, oneTimeClient=0, _events.size()=131822, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Tue May 21 18:48:36 2024 is using 157278423 bytes. Total tcpout queue size is 157286400. Warningcount=9001 That went from Warningcount=1 at 18:48:38.538 to Warningcount=1001 at 18:48:38.771 Then 18:48:38.90 has 2001 18:48:39.033 has 3001 18:48:39.134 has 4001 18:48:39.200 has 5001 18:48:39.336 has 6001 18:48:39.553 has 7001 18:48:46.500 has 8001 and finally: 18:48:47.099 has 9001 I suspect the backpressure is caused by an istio pod failure in K8s. I haven't tracked down the cause but I've seen some cases where the istio ingress gateways pods in K8s are in a "not ready" state, however I suspect they were alive enough to take on traffic. During this time period I will sometimes see higher than normal Warningcount= entries *and* often around the same time my website availability checks start failing to DNS names that are pointed to istio pods. My current suspect is that's it's not just a Splunk-level backpressure but I'll keep investigating (at the time the indexing tier shows the most utilised TCP input queues were at 67% using a max() measurement on their metrics.log. The vast majority of my Warningcount= entries on this forwarder show a value of 1. The configuration for this instance is: maxQueueSize = 150MB autoLBVolume = 10485760 autoLBFrequency = 1 dnsResolutionInterval = 259200 # tweaks the connectionsPerTarget = 2 * approx number of indexers connectionsPerTarget = 96 # As per NLB tuning heartbeatFrequency = 10 connectionTTL = 75 connectionTimeout = 10 autoLBFrequency = 1 maxSendQSize = 400000 # default 30 seconds, we can retry more quickly with istio as we should move to a new instance if it goes down backoffOnFailure = 5   The maxSendQSize was tuned for a much lower volume forwarder and I forgot to update it for this instance, so I will increase that, and this instance appears to have increased from 30-50MB/s to closer to 100MB/s so I'll increase the autoLBVolume setting as well ... View more

One minor request, if this logging is ever enhanced can it please include the output group name. 05-16-2024 03:18:05.992 +0000 WARN AutoLoadBalancedConnectionStrategy [85268 TcpOutEloop] - Current dest host connection <ip address>:9997, oneTimeClient=0, _events.size()=56156, _refCount=1, _waitingAckQ.size()=0, _supportsACK=0, _lastHBRecvTime=Thu May 16 03:18:03 2024 is using 31477941 bytes. Total tcpout queue size is 31457280. Warningcount=1001 Is helpful, however the destination IP happens to be istio (K8s software load balancer) and I have 3 indexer clusters with different DNS names on the same IP/port (the incoming DNS name determines which backend gets used). So my only way to "guess" the outputs.conf stanza involved is to set a unique queue size for each one so I can determine which indexer cluster / output stanza is having the high warning count. If it had tcpout=<stanzaname> or similar in the warning that would be very helpful for me. Thanks ... View more

Thankyou very much for the detailed reply, that gives me enough to action now. I appreciate the contributions to the community in this way. ... View more

Thanks, I'll review the maxQueueSize If the warning count was higher, such as 20 in your example. What would be the best way to determine a good value (in bytes) for maxSendQSize to avoid the slow indexer scenario? ... View more

This setting definitely looks useful for slow receivers, but how would I determine when to use it, and an appropriate value? For example you have mentioned: WARN AutoLoadBalancedConnectionStrategy [xxxx TcpOutEloop] - Current dest host connection nn.nn.nn.nnn:9997, oneTimeClient=0, _events.size()=20, _refCount=2, _waitingAckQ.size()=4, _supportsACK=1, _lastHBRecvTime=Thu Jan 20 11:07:43 2024 is using 20214400 bytes. Total tcpout queue size is 26214400. Warningcount=20  I note that you have Warningcount=20, a quick check in my environment shows Warningcount=1, if i'm just seeing the occasional warning I'm assuming tweaking this setting would be of minimal benefit?   Furthermore, how would I appropriately set the bytes value? I'm assuming it's per-pipeline, and the variables involved might relate to volume per-second per-pipline, any other variables? Any example of how this would be tuned and when?   Thanks ... View more

This looks very useful, is there a recommended way to set the maxSendQSize ? Do I need to vary it depending on the thruput of the HF per pipeline? I'm assuming the maxSendQSize would be in-memory buffer/queue per-pipeline in addition to the overall maxQueueSize? Finally I'm assuming this would be useful when there is no load balancer in front of the indexers? ... View more

I regularly have excess buckets and I haven't tested if they would freeze as per normal. I would assume they would as they are still tracked by the cluster manager...they are just extra buckets for the generation the manager has set, they are just considered excess to the current replication and search factor. Otherwise it's just another bucket.   I even have https://ideas.splunk.com/ideas/E-I-75 open as I believe they should automatically remove themselves over time.   ... View more

Ok I see it now. The previous setting was in server.confirm but this setting is in inputs.conf   thanks ... View more

Definitely an improvement, however the sslServerHandshakeTimeout is in server.conf so perhaps the answer should advise that? It looks like it refers currently to inputs.conf ... View more

Just on inputs.conf you have: inputs.conf [http] #(useful when HEC clients are using connection pools and want to keep connections idle. Set it 2 times expected idle time of connection. ) sslServerSessionTimeout = 300 However I found that sslServerSessionTimeout appears to be in server.conf Is that the wrong conf file? ... View more

I'm unaware of any limits here...I haven't seen a documented limit for Splunk cloud instances for REST APi calls. There may be an AWS system that is helping to prevent a flood of traffic...(I'm not familiar with the actual Splunk cloud build) ... View more

I built a version control app for allowing backup and restore of what users created but there are many approaches to this issue. The app is linked in my signature. Other options for backup or config management include those from community slack. Pasted below  Helpful links for configuration management and deployment automation: - https://pypi.org/project/kintyre-splunk-conf/ - https://github.com/Bre77/splunk-config-editor - https://github.com/paychex/Splunk.Conf19 - https://github.com/paychex/splunk-python/tree/main/Splunk2Git - https://splunkbase.splunk.com/app/4355/ and https://splunkbase.splunk.com/app/5061/ - https://github.com/atkouki/splunk-cloud-cicd - https://github.com/splunk/acs-privateapps-demo - https://splunkbase.splunk.com/app/6368/ (btool output, even for indexers) - https://github.com/gjanders/Splunk many scripts for managing KOs via REST API - Export custom apps (including local) and local config from splunkbase apps: https://splunkbase.splunk.com/app/6686   ... View more

Alerts for splunk admins has an example for historical searches using the introspection data. Refer to my signature for a link                                ... View more

So repeating what @isoutamo and @PickleRick have mentioned, don't mix volumes and non-volumes on a path, that will not end well. Additionally, when you size your volume, maxVolumeDataSizeMB , keep in mind that this is for the buckets. The DMA objects do *not count* towards this size, so if you have DMA you need to change the volume:_splunk_summaries as well or you may fill that disk with summaries. Another tip is that if you use dbinspect or similar to measure the size, I found in previous versions (7.x or prior) I saw a difference of around 25% between what Splunk measured and what I saw on the actual disk of a Linux server. Finally, this game changes completely for smartstore , in that case you just configure cache padding instead...(and/or you can use maxGlobalRawDataSizeMB per-index which is for the whole index, not per-indexer server). Since you are not using smartstore, maxTotalDataSizeMB is per index (which you could override in [default], but also *per* indexer, so you need to do the maths carefully. ... View more

While I'm not completely sure what the new endpoints might be, one thing you might try doing is doing an update on the UI and watching the splunkd_access.log file in $SPLUNK_HOME/var/log/splunk to see if you can tell which endpoints are used You might be able to determine the endpoint required to update the config from there... ... View more