Windows event logs—from Security auditing and Sysmon to PowerShell script blocks—form the operational backbone of modern Security Operations Centers (SOCs). However, in high-throughput environments, running hundreds of threat detections against raw XML event streams has historically come at a steep compute cost. Starting with the release of the [Splunk Add-on for Microsoft Windows 11.0.3]  ( https://splunkbase.splunk.com/app/742 ), we are introducing native Index-Time Auto-Extraction ( xmlkv-winevt ) and intelligent Hybrid Extraction. With just a single configuration toggle, security teams can dramatically accelerate search performance and unlock greater detection concurrency across Enterprise Security (ES) and ESCU content. Requires Splunk 10.4.3/10.6 and above on Splunk Platform side. The Challenge: The Search-Time REGEX Bottleneck In a traditional setup, Windows XML events are ingested as raw text and parsed at search time: Over 350+ Enterprise Security Content Update (ESCU) detections rely on search-time key-value extractions. The Splunk_TA_Windows app evaluates over 120 regular expressions per matching security event at search time (with 500+ REGEXes defined across the TA). Scanning wide time ranges or high-volume event streams forces search heads to spend significant CPU and memory repeatedly parsing raw XML text. This overhead introduces search latency, increases cluster compute pressure, and limits the number of real-time or scheduled detections an organization can reliably run.       Traditional Search-Time Flow: [ Ingest Raw XML ] ──> [ Search Dispatched ] ──> [ 120+ Regex Parsers per Event ] ──> [ High CPU / Latency ] The Solution: Index-Time Auto-Extraction & Hybrid Processing Splunk Add-on for Microsoft Windows 11.0.3 introduces native index-time extraction ( xmlkv-winevt ) tailored specifically for Windows XML event structures (e.g., EventID , Provider , EventData key-value pairs). Instead of relying on regex execution during search queries: Fields are pre-tokenized and indexed during ingestion. Search filters hit indexed tokens directly ( tsidx ), bypassing the need to parse raw text with regex. Smart Hybrid Optimization ( OPTIMIZE_IE_EXTRACT ) Splunk dynamically inspects search requirements. If all fields needed by a search are already indexed, expensive search-time regex extractions are automatically skipped.     Hybrid Index-Time Flow: [ Ingest & Auto-Extract ] ──> [ Fields Pre-Indexed ] ──> [ Search Dispatched ] ──> [ Fast tsidx Match & Zero Regex ] 100% Backward Compatible A primary design requirement for this feature was seamless enterprise adoption without breaking existing data pipelines or detection logic: Historical Events: Events indexed prior to enabling index-time extractions continue to work seamlessly via standard search-time fallback regexes. Mixed Data Streams: If a detection query requests a custom or long-tail field that was not extracted at index time, Splunk automatically falls back to search-time extraction for that specific field. Zero Query Rewrites: Existing detection rules, macros, and datamodels require zero changes. Getting Started: Enabling Index-Time Extraction For Splunk Enterprise Security customers, the required configurations are bundled directly inside the [Splunk Add-on for Microsoft Windows (v11.0.3+)] ( https://splunkbase.splunk.com/app/742 ). 1. Upgrade Prerequisites Splunk Platform: Version 10.4.3/10.6 or later (Splunk Enterprise or Splunk Cloud Platform). Technology Add-on: Install or upgrade to [Splunk Add-on for Microsoft Windows 11.0.3] ( https://splunkbase.splunk.com/app/742 ) on your Search Heads and indexers( indexers need props.conf and transforms.conf for index time config). 2. The One Configuration Change The extraction mode definition ( INDEXED_EXTRACTIONS = xmlkv-winevt ) is pre-packaged in the TA. To enable hybrid extraction, activate the typing pipeline in your local props.conf : [source::XmlWinEventLog:*] XML_INDEXED_EXTRACTIONS_PIPELINE = typing Advanced Ingest Tuning & Options ( optional) Splunk 10.6 provides granular controls in props.conf to balance tsidx storage size and search performance according to the official Splunk documentation: Observed Performance & Operational Impact Metric Search-Time Extraction Index-Time Hybrid (Splunk 10.4.3/10.6 + TA 11.0.3) Search Latency / Run Time High (scales with event volume) Significantly Reduced (hits tsidx directly) Search Head CPU & Memory High (120+ REGEX evaluations/event) Minimal (Search-time REGEX skipped) Detection Concurrency Constrained by CPU pressure Higher detection throughput across ESCU Storage Impact Baseline Modest increase in tsidx size Backward Compatibility Standard Full seamless fallback for historical data Next Steps Review the props.conf documentation bundled inside the [Splunk Add-on for Microsoft Windows 11.0.3] ( https://splunkbase.splunk.com/app/742 ). Test the configuration in a staging environment by setting XML_INDEXED_EXTRACTIONS_PIPELINE = typing and comparing search run times on identical ESCU detection searches. Deploy to production indexers/forwarders to scale your Windows security detection operations. Retire Brittle Pre-Ingest Conversions: If your organization relies on custom/hacky workarounds to convert XML into JSON before sending to Splunk, you risk losing critical security keys or having events rejected by downstream indexer policies. Splunk’s out-of-the-box (OOTB) hybrid solution natively knows when an event or field is not eligible for index-time extraction and gracefully falls back to search-time extraction—ensuring zero data loss and complete event fidelity. ... View more

Why Universal Forwarders Should Not Be Used as Intermediate Forwarders A practical Splunk forwarding topology guide for avoiding event corruption, duplicate data, and blocked forwarding queues. If you are designing a Splunk forwarding architecture, it can be tempting to place a Universal Forwarder (UF) in the middle of an ingest path and use it as a relay. In some environments, this looks simple: one forwarder tier receives data, then forwards it to indexers. However, not every intermediate-forwarder pattern is safe. The most important distinction is whether the upstream sender is a Universal Forwarder or a Heavy Forwarder. Short answer: Use a Heavy Forwarder as the intermediate hop when data originates from a Universal Forwarder. Avoid UF → IUF → IDX for production ingest paths. Quick recommendation     Topology Recommendation Summary UF → IHF → IDX OK Recommended when a UF needs to forward through an intermediate tier. UF → IUF → IDX Not OK Avoid. This can expose downstream systems to partial-event behavior and hard-to-diagnose ingest problems. HF → IUF → IDX OK Acceptable because the upstream Heavy Forwarder performs parsing before the data reaches the intermediate UF.       Topology diagram     Pattern Flow Result Preferred for UF sources UF → IHF → IDX OK Avoid for UF sources UF → IUF → IDX Not OK Acceptable when source is HF HF → IUF → IDX OK       Legend: UF = Universal Forwarder, HF = Heavy Forwarder, IHF = Intermediate Heavy Forwarder, IUF = Intermediate Universal Forwarder, IDX = Indexer. Why UF → IHF → IDX is OK When a Universal Forwarder sends data to an intermediate Heavy Forwarder, the Heavy Forwarder provides a parsing layer that helps contain partial events before they continue downstream. If an upstream UF disconnects, restarts, or breaks the connection while an event is still incomplete, the intermediate HF can prevent that partial event from being forwarded deeper into the ingest pipeline. This reduces the risk of downstream event corruption and queue problems. Practical benefit: The intermediate HF gives Splunk a safer place to handle incomplete event data before it reaches indexers. Why UF → IUF → IDX is not OK An intermediate Universal Forwarder is not designed to safely parse, repair, or discard partial events received from another Universal Forwarder. It primarily forwards the stream onward. That means a partial event fragment can pass through the intermediate UF and reach the downstream indexer. Once that happens, the downstream system may be left trying to reconcile incomplete data with later data from the same source. Possible symptoms include: Wrongly merged events or data munging Missing events Repeated or duplicated events Permanently blocked tcpout queues Blocked replication queues or stalled throughput caused by pipeline channel corruption Example failure sequence A source UF sends a partial block to the intermediate UF. In this context, a partial block means the UF has transmitted a chunk of data where the last event in that chunk does not yet contain its event boundary. The IUF receives bytes that look like normal forwarded data, but it cannot know whether the final event in the block is complete because it is not doing the parsing and event-boundary handling that a Heavy Forwarder or indexer would perform. Common ways a UF can end up sending only the beginning of a block include: AWS Auto Scaling scale-down: UF instances running on AWS may be terminated during a scale-down event. If the instance is stopped ungracefully while the UF is actively sending data, the TCP connection can disappear after only part of the current block has been sent. forceTimebasedAutoLB=true: With time-based automatic load balancing enabled, the UF can rotate or disconnect from the current IUF on schedule. If that rotation happens after the UF has already sent the beginning of a block but before the final event boundary is included, the IUF has already received a partial block. UF restart: A manual, scripted, or deployment-triggered restart can interrupt an active forwarding stream before the UF finishes sending the complete block. UF crash: If the UF process crashes, it can terminate the connection immediately and leave the last transmitted block incomplete from the receiver’s perspective. Network interruption: Any network issue between the UF and IUF, such as a transient outage, firewall reset, load balancer timeout, or routing flap, can break the connection while a block is still in flight. In all of these cases, the problem is not that the UF intentionally creates a bad event. The problem is that the connection can end after bytes for the next event have started flowing but before the event boundary is reached. An intermediate UF relays what it received; it does not parse and safely discard the incomplete trailing event before forwarding downstream. The intermediate UF forwards that partial data onward. The downstream indexer keeps the incomplete fragment in its parsing queue. Later, the remaining data arrives through a different path or at a different time. The downstream system may merge the old fragment with an unrelated later event. Key point: Once a partial event has passed through an intermediate UF, the UF has not provided the parsing protection needed to safely contain the problem. Configuring EVENT_BREAKER on the edge UF can reduce the chance of corruption in some cases, but it should not be treated as equivalent to using an intermediate Heavy Forwarder. Why HF → IUF → IDX is OK In the HF → IUF → IDX pattern, the upstream sender is a Heavy Forwarder. Because the HF performs parsing before forwarding, the intermediate UF is not receiving the same kind of raw partial-event stream that makes UF → IUF risky. In other words, the unsafe pattern is not simply “a UF appears in the middle.” The unsafe pattern is specifically when data from a UF is forwarded through another UF as an intermediate hop. Decision guide If the original sender is a UF and an intermediate hop is required, use an HF as the intermediate forwarder. If possible, send directly from UF → IDX without an unnecessary relay tier. If the original sender is an HF, an intermediate UF can be acceptable because parsing has already occurred upstream. Avoid placing a UF behind another UF as a production intermediate forwarding tier. Final takeaway Safe: UF → IHF → IDX Unsafe: UF → IUF → IDX Safe: HF → IUF → IDX If you need an intermediate forwarding tier for data that originates from Universal Forwarders, use Heavy Forwarders for that intermediate role. This design helps prevent partial-event handling problems from becoming downstream indexing, queue, or data-quality incidents. References https://community.splunk.com/t5/Getting-Data-In/Wrongly-merged-Events-permanently-blocked-tcpout-queue-with/m-p/508746 ... View more

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in recent releases. It solves a common operational problem: what happens when forwarding must continue, but the receiving tier is unavailable or unreachable for an extended period? Instead of depending on external queueing systems, the forwarder can now persist outbound events locally on disk and resume transmission when connectivity returns. Why this feature stands out: it addresses real outages, real maintenance windows, and real disconnected environments with a simple native control in outputs.conf . What it is TcpOut Persistent Queue allows a forwarder or intermediate heavy forwarder to write outbound data to disk when the destination cannot accept data immediately. When the receiving side becomes available again, the queued data is sent onward without requiring a third-party buffer such as SQS or S3. In practical terms, this gives Splunk administrators a native, local, and operationally simpler buffering option for disruptions that are temporary but too long for in-memory queues alone. Why it matters in the real world The strongest value of this feature is not theoretical architecture elegance. It is operational survival. Ships and cruises at sea: a vessel can lose internet or satellite connectivity for days. Logs and telemetry still continue to generate onboard. With tcpout persistent queue, that data can be retained locally and forwarded once connectivity returns. Corporate laptops: systems may stay off VPN or off network for extended periods. Events can be preserved and sent later when the endpoint reconnects. Indexer clustering maintenance: rolling restarts, planned upgrades, and temporary cluster-side slowness no longer have to immediately translate into data loss or emergency queue workarounds. Sensitive environments: data can remain within the managed network boundary instead of being detoured through external services. Multiple tcpout destinations: if one destination is down but has a sufficiently large persistent queue, another active destination can continue processing until the failed path's queue fills. Best example: disconnected maritime environments are an ideal use case. A ship continues to produce operational, security, and observability data while offline, then automatically drains the backlog when the link is restored. Data flow during outage and recovery The following diagram shows the most practical flow using the ships or cruises scenario.   Important caution: If the backlog drains from disk faster than tcpout can send over the network, memory can spike. Until fixed versions are in place, size PQ conservatively relative to available RAM. Why native PQ is often better than external buffering Before this capability, some teams considered third-party buffering patterns for long disconnections. Native tcpout PQ changes that calculation significantly. No additional subscription cost for external queueing services. No permanent detour latency introduced by writing every event out to an external queue first. No dependency on payload size constraints imposed by third-party services. No extra upload and download bandwidth for every event. No dependency on internet connectivity for buffering, because third-party options such as SQS or S3 still require an available network path to the cloud before they can store data. Reduced operational complexity compared with maintaining external queueing, batching, retry logic, and dead-letter handling. Better data locality for environments where sensitive data should remain internal. This does not mean external architectures never have a place. It means many common Splunk forwarding scenarios can now be solved more cleanly inside the platform. How to enable it Enable the feature by setting persistentQueueSize in outputs.conf for the tcpout target group.     [tcpout:splunk-group1] persistentQueueSize=1TB [tcpout:splunk-group2] persistentQueueSize=2TB This simplicity is part of the appeal. The feature is easy to understand and easy to turn on, provided sizing is thought through carefully. Sizing guidance A practical sizing approach is to estimate how much data a forwarding tier sends during the outage window you need to survive. One recommended method is to calculate this from internal metrics.     index=_internal source=*metrics.log* group=tcpin_connections hostname=<all IHF> host=<all idx> | stats sum(kb) as required_pq_in_kb by hostname Run the search across the number of hours or days you want to protect. The resulting required_pq_in_kb is the approximate overall PQ needed for a given intermediate heavy forwarder. A practical approximation from the source material is:     persistentQueueSize = required_pq_in_kb / parallelIngestionPipelines This provides a starting point, not a final truth. Real sizing should also consider growth bursts, recovery behavior, disk availability, and memory safety. Critical operational update: the PQ drain memory issue Important: an issue was identified where tcpout can read from the persistent queue at unbounded speed after the indexing tier becomes available, even though it cannot push events over TCP at the same rate. This can cause memory spikes and potentially an out-of-memory condition. The practical implication is straightforward: a very large persistent queue is not automatically a safe persistent queue. Example from the source material: If a system has 12 GB RAM and persistentQueueSize=18GB , it is likely to hit OOM if the queue fills or holds more than 12 GB of data. If the same system has 12 GB RAM and persistentQueueSize=8GB or less, it is much less likely to hit the issue. Interim guidance: until fixed versions are installed, keep persistentQueueSize below the average available memory of the system. Fix is available in versions 9.4.13, 10.0.7, 10.2.4, and 10.4.1. Practical design guidance Use tcpout PQ for real disconnect scenarios, not as a substitute for weak capacity planning. Size for the outage you truly expect, especially in ships, offshore systems, mobile fleets, and remote sites. Validate available disk and memory together; disk capacity alone is not enough. Test recovery behavior by simulating outage and reconnection in a lower environment. Review parallel ingestion and queue automation settings on newer Splunk versions where queue behavior is increasingly auto-adjusted. Be conservative on heavy forwarders with bursty HEC input, because sudden backlog release can amplify memory pressure. Related tuning context Persistent queueing at tcpout is most useful when viewed as part of a broader ingestion resilience strategy. Related operational guidance from internal documentation emphasizes that newer Splunk releases increasingly automate indexing and queue behavior, including settings such as autoAdjustQueue=true and queue sizing improvements on modern versions. For on-prem environments experiencing full replication or ingestion pressure, queue and pipeline tuning may still matter. However, tcpout PQ is especially valuable on the forwarding side because it provides a native answer to sudden spikes, disconnected operation, and upstream outage windows. When this feature is the right fit       Scenario   Why tcpout PQ helps   Main design concern   Ships, cruises, offshore systems Stores data locally during multi-day connectivity loss and forwards later Disk sizing for outage duration and safe drain after reconnection Corporate laptops off VPN Retains events while endpoints are disconnected Endpoint storage limits and intermittent shutdown behavior Indexer rolling restart or upgrade Buffers forwarding traffic during planned receiver unavailability Recovery surge after the receiving tier returns Temporary indexing slowness Prevents immediate forwarding blockage during short-term backpressure PQ should not hide chronic capacity shortfalls       Bottom line TcpOut Persistent Queue is one of Splunk's most practical features because it solves a problem administrators face constantly: data does not stop just because connectivity does. For maritime systems, remote environments, laptops, and planned indexer-side outages, native outbound persistence can dramatically simplify architecture and improve resilience. The feature is powerful, easy to enable, and operationally meaningful. The one caution is equally important: size it responsibly, especially on versions affected by the drain-related memory spike issue. Recommended approach: enable the feature for clearly defined outage scenarios, size it from observed throughput, test drain behavior, and patch to versions that address the memory risk.   Operational checklist Identify forwarders that experience planned or unplanned disconnection Estimate required backlog duration in hours or days Size persistentQueueSize from observed data volume Validate disk capacity and available memory together Simulate outage and reconnection before broad rollout Upgrade to fixed versions where applicable References New Splunk TcpOutput persistent queue Important update about TcpOut PQ size issue ... View more

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully utilized. In practice, that often leads to a familiar pattern: blocked queues appear, support investigates, administrators tune pipeline-related settings manually, and a restart is required before seeing whether the change helped. This can turn what should be a runtime adaptation problem into a long operational cycle. Auto ingestion pipeline scaling was introduced to reduce that friction. Instead of relying only on a fixed parallelIngestionPipelines value, Splunk can increase the number of ingestion pipeline sets dynamically when demand rises and sufficient resources are available. The goal is simple: make better use of available hardware, reduce snowflake tuning, and respond faster to bursty or growing ingestion workloads. Think of this feature as letting splunkd scale ingestion parallelism at runtime instead of requiring administrators to pre-pick a permanent pipeline count for every workload profile. What auto ingestion pipeline scaling does At a high level, the feature allows Splunk to automatically scale pipeline sets upward in response to ingestion demand. Internal project material describes the broader objective as auto-scaling pipelines, channels, and related ingestion behavior to eliminate manual tuning and reduce long-standing operational issues. Automatic scale-up of pipeline sets: Splunk can increase the number of ingestion pipeline sets when blocked-queue conditions and available resources indicate more parallelism would help. No restart needed for runtime scaling: A major benefit called out in internal references is eliminating restart-driven tuning for pipeline configuration changes. Better hardware utilization: The feature was created in part to address environments that were seeing ingestion errors or blocked conditions even while substantial CPU capacity remained unused. Broader operational simplification: The associated initiative aimed to eliminate one-off “magic number” tuning across customer environments. This feature as available from Splunk Enterprise 9.4 for on-premises environments, with later improvements such as scale-down behavior noted for 10.2.0. How to think about the setting The key administrative switch is:     [general] pipelineSetAutoScale = true   When enabled, Splunk is allowed to scale pipeline sets automatically rather than relying solely on the fixed value of parallelIngestionPipelines. According to internal documentation and discussion, the feature uses runtime conditions to decide whether scale-up is appropriate, and it differentiates between deployment roles such as intermediate/universal/heavy forwarders and indexers. This is important because a common misunderstanding is to treat pipeline autoscaling as a blanket “more pipelines is always better” switch. That is not the right model. The feature is intended to increase ingestion parallelism selectively when both demand exists and the host has the CPU and I/O headroom to support it. When it helps most Auto ingestion pipeline scaling is most useful in environments where ingestion pressure exists but the infrastructure still has headroom. High-ingestion tiers with spare CPU and memory Intermediate or heavy forwarders On-prem indexing tiers Cases where blocked queues occur because pipeline parallelism is too low, not because storage is saturated One of the internal drivers for this work came from customer scenarios where HEC ingestion was encountering pressure and 503 responses despite hardware utilization remaining below expected limits. Guidance on the following exclusions: Splunk Cloud Victoria / VE indexers that pull blobs from SmartBus Nodes with Native Ingest for Amazon Security Lake enabled Remote-input-enabled nodes where data bypasses the standard Splunk ingestion pipeline path and ingested to SmartBus Equivalent on-prem architectures that similarly detour data away from the normal ingestion pipeline to SmartBus The reason is straightforward: if the standard ingestion pipeline is not the limiting path, scaling pipeline sets will not address the actual bottleneck and may create confusion about what the feature can or cannot improve. Operational prerequisites before enabling Enable pipeline autoscaling only when the tier has meaningful unused resources. In other words, this feature is best used to unlock unused capacity, not to compensate for an already exhausted system.     Check Recommended guidance General suitability Use when pipeline queues are full and the tier still has meaningful CPU and memory headroom. Highly recommended tiers IHF, IUF, ingest tier, and on-prem indexers with at least 50% CPU and memory available. Search head and indexing tier minimums Before enabling on SH or indexing tier, ensure at least 40% memory available, at least 25% CPU available, and load average below 1.00 where specified.   Important caveat: more pipelines can increase downstream pressure One of the most useful discussions on this topic addresses a real concern: if Splunk increases ingestion pipeline sets, what happens to per-pipeline behaviors such as hot bucket creation and storage demand? The answer is that more pipelines can increase concurrency. For indexers, settings like maxHotBuckets apply per pipeline. That means raising pipeline count can increase the total number of active hot buckets across the instance. In environments where storage IOPS is already the bottleneck, simply increasing pipeline parallelism manually can make things worse rather than better. That is why this autoscaling feature logic is intended to avoid scaling up when I/O is already a problem, but administrators should still treat storage validation as part of rollout planning. Practical takeaway: if your indexers are blocked because disks cannot keep up, adding more ingestion parallelism is not the first fix. Validate whether the constraint is CPU-bound pipeline throughput or I/O-bound indexing throughput. Does it scale back down? Yes, but version context matters. The scale-down support was added for 10.2.0. That matters because administrators naturally want to know whether a temporary surge leaves the instance permanently scaled up. So the safe summary is: Scale-up is a core part of the feature introduced earlier. Scale-down was added from 10.2.0. Best practices Enable it only where the standard Splunk ingestion pipeline is actually in use. Check CPU, memory, and load headroom before rollout. Use it to unlock unused capacity, not to mask storage or architecture constraints. Do not enable it on Victoria/VE indexing tiers or similar SmartBus-detour architectures. Document version expectations clearly for scale-up versus scale-down behavior. Suggested rollout pattern Confirm the workload uses the standard ingestion pipeline path. Exclude Victoria/VE indexers, ASL-enabled nodes, and remote-input-detour architectures. Measure CPU, memory, and load headroom using introspection data. Verify blocked conditions are not primarily caused by storage I/O. Enable pipelineSetAutoScale = true on an appropriate tier. Observe ingestion performance, blocked queue trends, and system resource utilization. Expand gradually to similar tiers once behavior is validated. Bottom line Auto ingestion pipeline scaling is best understood as a runtime parallelism feature for Splunk ingestion. It helps Splunk respond to demand by increasing pipeline sets when the system has the available resources to benefit. Used in the right place, it reduces manual tuning, avoids restart-driven adjustments, and makes better use of existing hardware. But it is not a universal switch for every deployment. It should not be enabled on Splunk Cloud Victoria indexing tiers or similar architectures where data detours away from the standard ingestion pipeline, and it should not be used as a substitute for resolving storage or non-pipeline bottlenecks. If you treat it as a targeted way to scale ingestion parallelism where Splunk still has real headroom, you will get the most value from it. References Best Practices: Splunk auto adjust pipeline queue Splunk auto-scalable ingestion pipelines discussion   ... View more

Splunk asynchronous forwarding is often misunderstood as simply setting autoLBVolume. That is not quite right. The key point is that asynchronous forwarding became possible because the behavior behind autoLBVolume was redesigned starting in Splunk 7.3.6. Before that release, autoLBVolume still existed, but it operated using a synchronous forwarding model. Important clarification: autoLBVolume is a setting. Asynchronous forwarding is the forwarding behavior enabled by the newer design of that setting in supported versions. What changed in Splunk 7.3.6 Before Splunk 7.3.6, load-balanced forwarding using autoLBVolume was synchronous. A forwarder would continue sending to its current target in a more serialized way, which meant slower or unhealthy receivers could delay overall forwarding behavior. Starting with Splunk 7.3.6, autoLBVolume was redesigned to support asynchronous forwarding. This allowed the forwarder to move data more efficiently across multiple receiving indexers or intermediate forwarders, reducing the chance that one slow destination would stall ingestion flow. Why this matters Better data distribution: Data is spread more evenly across receivers. Higher ingestion throughput: Forwarders can push data faster in high-volume environments. Less impact from slow receivers: A few overloaded indexers are less likely to pause or bottleneck the pipeline. Lower intermediate-tier cost: Efficient forwarding can reduce pressure on heavy forwarders used as a routing layer. Practical outcome: In busy environments, asynchronous forwarding can improve both indexing performance and downstream search performance by keeping bucket and event distribution more balanced. Common misconception A common statement is: “Asynchronous forwarding is autoLBVolume.” A more accurate statement is: autoLBVolume is the mechanism used for load-based forwarding, and from Splunk 7.3.6 onward that mechanism operates asynchronously. So if someone says “enable async by configuring autoLBVolume,” that is directionally useful, but technically incomplete. The version matters. On versions earlier than 7.3.6, configuring autoLBVolume does not give you the modern asynchronous behavior. Do not assume feature parity across versions: the same setting name can behave differently depending on the Splunk release. Version-aware summary Splunk version autoLBVolume behavior Forwarding model Before 7.3.6 Classic implementation Synchronous 7.3.6 and later Redesigned implementation Asynchronous Later 9.x releases Matured support with broader compatibility Asynchronous   Configuration guidance When using asynchronous forwarding, autoLBVolume should be sized based on observed throughput, and maxQueueSize should be large enough to support efficient buffering. Connection behavior also matters, especially if acknowledgement is enabled. With ACK: use larger queue sizing and longer connection lifetime. Without ACK: use a shorter connection lifetime aligned to receiver behavior. Avoid mixing with time-based load balancing: forceTimebasedAutoLB=false is important for async behavior. Hard rule: asynchronous forwarding does not work with forceTimebasedAutoLB. If that setting is enabled, you are not using the intended async forwarding behavior. How to configure asynchronous forwarding for Splunk 9.x and above Use the following examples for Splunk Heavy Forwarder 9.x and above. Size autoLBVolume from the forwarder throughput in metrics.log, and size maxQueueSize at least five times larger than autoLBVolume. Without acknowledgement [tcpout:XXXXXXXX] autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 10MB) # connectionTTL = Less than receiver shutdown timeout (90 default) connectionTTL = 75 connectionTimeout = 10 forceTimebasedAutoLB = false   With acknowledgement [tcpout:XXXXXXXX] autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 20MB) connectionTTL = 300 useACK = true connectionTimeout = 10 forceTimebasedAutoLB = false     How to configure asynchronous forwarding for Splunk 7.3.6 to 8.x Use the following examples for Splunk Heavy Forwarder 7.3.6 through 8.x. In these versions, set heartbeatFrequency based on whether acknowledgement is enabled. This setting is not required for Splunk IUF/IHF 9.0 onward. With acknowledgement [tcpout:XXXXXXXX] forceTimebasedAutoLB = false autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 20MB) connectionTTL = 300 useACK = true # heartbeatFrequency Must be less than writeTimeout. Not needed for Splunk IUF/IHF 9.0 onwards heartbeatFrequency = 15 Without acknowledgement [tcpout:XXXXXXXX] forceTimebasedAutoLB = false autoLBVolume = <metrics.log average_kbps> * 1000 (minimum 512000) maxQueueSize = (5 or more) x autoLBVolume (minimum 10MB) #connectionTTL = Less than receiver shutdown timeout (90 default) connectionTTL = 75 # heartbeatFrequency Must be greater than writeTimeout. Not needed for Splunk IUF/IHF 9.0 onward. heartbeatFrequency = 350 Version note: For Splunk 7.3.6 to 8.x, configure heartbeatFrequency explicitly. For Splunk IUF/IHF 9.0 onward, this setting is no longer required in this async forwarding configuration. How to calculate autoLBVolume Calculate autoLBVolume from the average throughput reported in metrics.log. For multiple forwarders, run the SPL across all forwarder hosts, convert average_kbps into bytes, and then average the result.   index=_internal sourcetype=splunkd component=metrics earliest=-24h group=thruput name=thruput host=<all fwd hosts> | eval autoLBVolume = average_kbps * 1024 | stats avg(autoLBVolume)   Rule of thumb: Use the calculated value for autoLBVolume, but do not set it below the minimum recommended value of 512000. Then size maxQueueSize to at least five times autoLBVolume, using the higher minimum for ACK-enabled configurations. Note: For asynchronous forwarding, keep forceTimebasedAutoLB=false. Do not enable time-based load balancing when using this async forwarding configuration. Short version If you want the simplest accurate explanation: autoLBVolume existed before 7.3.6. Before 7.3.6, it was synchronous. From 7.3.6 onward, it was redesigned to be asynchronous. So asynchronous forwarding is not just the setting name; it is the post-7.3.6 behavior of that setting. Reference Read the LinkedIn post ... View more

When a Splunk forwarder loses connectivity to its indexers, it does not always reconnect immediately. In many environments, this looks like a mysterious delay after the network comes back: the path is healthy again, but forwarding resumes only after a waiting period. The reason is backoff. By default, the maximum backoff can reach 300 seconds because the default autoLBFrequency is 30, and active backoff is calculated as 1-10 * autoLBFrequency. If your goal is to make forwarders retry almost immediately after a full network outage, the key is not an old backoff toggle. The effective control is autoLBFrequencyIntervalOnGroupFailure. What causes the delay? Forwarders apply backoff after repeated write failures to an indexer. The related settings are: maxFailuresPerInterval = 2 secsInFailureInterval = 1   That means if the forwarder sees more than two failures within one second, it enters backoff. The important part is how long that backoff lasts. 1-10 * autoLBFrequency This is the active behavior. The backoff starts at 1 * autoLBFrequency, then increases on subsequent failures to 2 * autoLBFrequency, and keeps growing until it is capped at 10 * autoLBFrequency. With the default value autoLBFrequency = 30, the progression can rise to a maximum of 300 seconds. Initial backoff: 30 seconds Next backoff: 60 seconds Maximum backoff: 300 seconds If you already use a lower value, the cap is lower too. For example: autoLBFrequency = 10 → maximum backoff is 100 seconds autoLBFrequency = 5 → maximum backoff is 50 seconds Can a forwarder skip backoff entirely? Do not use backoffOnFailure for this purpose. It is an unused configuration and no longer active. So if the question is, “Can I disable backoff with backoffOnFailure?” the answer is no. Ignore that setting. In practice, the forwarder’s retry behavior is driven by the active backoff logic tied to autoLBFrequency, unless group-failure behavior overrides it. The setting that matters for fast recovery If the entire target group is unreachable, you can make the forwarder retry much faster by setting: autoLBFrequencyIntervalOnGroupFailure = <integer>   This setting changes how often the forwarder attempts to reconnect when no target in the tcpout group is reachable. autoLBFrequencyIntervalOnGroupFailure = <integer> * When the entire target group is not reachable, 'autoLBFrequencyIntervalOnGroupFailure' is the amount of time, in seconds, that a forwarder waits before attempting to connect to a target host in the group. * While 'autoLBFrequencyIntervalOnGroupFailure' is in effect, 'autoLBFrequency' is ignored. Once first connection is established to a group, 'autoLBFrequency' comes into effect again. * This setting is applied only when 'autoLBFrequencyIntervalOnGroupFailure' is less than 'autoLBFrequency'. * Every 'autoLBFrequencyIntervalOnGroupFailure' seconds, a new indexer is selected randomly from the list of indexers provided in the server setting of the target group stanza. * -1 means this setting is not active. * Default: -1   Best practice: If you want near-immediate recovery after a full outage, set autoLBFrequencyIntervalOnGroupFailure to a very low value such as 1. How it works in real life autoLBFrequency and autoLBFrequencyIntervalOnGroupFailure are independent, but when the full tcpout group is down, the lower applicable setting governs retry timing. Here is the practical behavior: If the whole group is unreachable, autoLBFrequencyIntervalOnGroupFailure can take over While it is active, autoLBFrequency is ignored Once one connection in the group succeeds, normal autoLBFrequency behavior resumes This means you can preserve your normal load-balancing cadence while still making outage recovery aggressive. Recommended configuration If you want the forwarder to retry every second after the entire destination group goes down, configure: [tcpout] autoLBFrequency = 30 autoLBFrequencyIntervalOnGroupFailure = 1 maxFailuresPerInterval = 2 secsInFailureInterval = 1   With this setup: Normal forwarding behavior still uses autoLBFrequency = 30 If the entire tcpout group becomes unreachable, the forwarder retries every 1 second You do not need to reduce autoLBFrequency just to improve outage recovery Outcome: Setting autoLBFrequencyIntervalOnGroupFailure = 1 effectively reduces maximum wait time after full group failure to about 1 second, without changing your regular autoLBFrequency. What to remember The default backoff behavior is tied to autoLBFrequency Default maximum backoff is 300 seconds because 10 * 30 = 300 backoffOnFailure is unused and should be ignored For fast reconnect after a complete outage, use autoLBFrequencyIntervalOnGroupFailure If you set autoLBFrequencyIntervalOnGroupFailure = 1, the forwarder can retry every second when the whole tcpout group is unavailable If your goal is “reconnect immediately after the network comes back,” this is the cleanest approach: keep your normal load-balancing settings, and add a low autoLBFrequencyIntervalOnGroupFailure so the forwarder probes the target group aggressively during full outage conditions. ... View more

When you enable autoAdjustQueue in Splunk, maxSize should be understood as the queue size Splunk starts with before automatic scaling takes over. It is not a memory guard rail and it is not the final limit Splunk can use when the feature is active. With autoAdjustQueue = true, Splunk begins auto scaling only after the configured queue size is full. In other words, maxSize acts as the minimum starting size for the queue. Once the queue reaches that size, Splunk can adjust the queue dynamically based on runtime conditions. Core guidance: If you enable autoAdjustQueue, set maxSize to a reasonable minimum starting size. A common practical range is 10MB to 100MB. This gives the queue enough initial capacity while still allowing Splunk to scale it automatically when the queue becomes full. Why an oversized maxSize is a bad idea Consider this configuration: [queue] maxSize = 500GB autoAdjustQueue = true   Since maxSize is the minimum starting size when autoAdjustQueue = true, Splunk will attempt to use a 500GB queue from the start, before automatic scaling has any useful role to play. Automatic scaling only becomes relevant after the configured queue size is full. With a 500GB starting point, the queue may never fill under normal workloads, which means: autoAdjustQueue becomes effectively useless because the precondition for scaling is never reached. There is a real risk of out-of-memory pressure if the host cannot support such a large queue allocation. If the queue does fill at 500GB, automatic scaling may attempt to grow it further, increasing memory pressure instead of protecting the instance. Bottom line: A massive hard-coded maxSize defeats the purpose of autoAdjustQueue. Use a practical starting size, such as 10MB to 100MB, and let splunkd scale the queue based on actual runtime conditions. Why maxSize is not the memory guard rail A common misunderstanding is to treat maxSize as the maximum amount of memory the queue can ever consume. That interpretation is misleading when autoAdjustQueue is enabled. In this mode, maxSize defines the point at which automatic queue adjustment begins. If the queue does not fill, Splunk has no reason to grow it. Once the queue reaches the configured size, splunkd can start increasing the queue size dynamically. Consider this example: [queue] maxSize = 50MB autoAdjustQueue = true   In this configuration, the queue starts with a 50MB size. When that queue fills, autoAdjustQueue allows Splunk to scale the queue beyond that starting point as needed. Important: Do not describe maxSize as the safety ceiling when autoAdjustQueue is enabled. It is better described as the minimum configured queue size that triggers automatic scaling once full. What autoAdjustQueue is supposed to do autoAdjustQueue helps Splunk respond to queue pressure dynamically. Instead of relying only on a fixed queue size chosen in advance, Splunk can expand queue capacity when the configured queue becomes full. That means the right approach is to: Enable autoAdjustQueue when you want Splunk to manage queue growth dynamically Set maxSize as a practical minimum starting size, not as a memory cap Allow splunkd to scale the queue after the configured size is reached Good queue tuning is not about using maxSize as a memory protection boundary. It is about choosing a sensible starting size and letting Splunk scale when queue pressure requires it. Recommended maxSize range A practical best practice is to keep maxSize between 10MB and 100MB when autoAdjustQueue is enabled. This range works well because it provides: A reasonable starting point: The queue has enough space for normal buffering before auto scaling is needed Predictable behavior: Auto scaling begins only when the configured queue becomes full Operational simplicity: Administrators do not need to over-size queues manually in an attempt to handle every possible spike Best practice: Start with a conservative maxSize in the 10MB to 100MB range and let splunkd grow the queue automatically after that configured size fills. Recommended configuration pattern     Approach Example Expected behavior Recommended maxSize = 50MB autoAdjustQueue = true Queue starts at 50MB and can auto scale after it becomes full Acceptable maxSize = 100MB autoAdjustQueue = true Queue starts at a larger, still reasonable size before automatic scaling begins Usually unnecessary maxSize = 500GB autoAdjustQueue = true Creates an extremely large initial queue target and delays the point at which auto scaling behavior becomes relevant   Practical takeaway If you use autoAdjustQueue, configure maxSize as the minimum queue size you want Splunk to fill before automatic scaling begins. Avoid framing it as a memory guard rail, because that is not how the setting behaves in this configuration. The short version: maxSize is the minimum configured queue size when autoAdjustQueue = true Auto scaling starts once the queue reaches that configured size Use 10MB to 100MB as a practical starting range Let splunkd scale the queue dynamically after it becomes full Bottom line: With autoAdjustQueue = true, maxSize should be treated as the queue’s minimum starting size. Auto scaling begins once that queue is full. Note:  There are specific configurations where autoAdjustQueue is automatically disabled or has no effect. Be aware of the following cases: a) maxKbps disables autoAdjustQueue If ingestion throttling is configured via maxKbps , autoAdjustQueue will disable itself. You will see the following warning in splunkd logs:       Ingestion throttling (maxKBps) is set disabling autoAdjustQueue Recommendation: Do not combine maxKbps with autoAdjustQueue . If you need autoAdjustQueue to function, set  maxKbps=0 . b) tcpout queue will not auto-scale if any of the following is true useACK = true — Acknowledgement-based forwarding prevents the tcpout queue from auto-scaling. tcpout persistent queue is enabled — When a persistent queue is configured for tcpout, auto-scaling is bypassed. In either case, autoAdjustQueue will have no effect on the tcpout queue, and you should size it manually. ... View more

Forwarder applies backoff setting (default 300 sec) based on following default settings ####### # Backoff Settings When Unable To Send Events to Indexer # The settings in this section determine forwarding behavior when there are # repeated failures in sending events to an indexer ("sending failures"). ####### maxFailuresPerInterval = <integer> * The maximum number of failures allowed per interval before a forwarder applies backoff (stops sending events to the indexer for a specified number of seconds). The interval is defined in the 'secsInFailureInterval' setting. * Default: 2 secsInFailureInterval = <integer> * The number of seconds contained in a failure interval. * If the number of write failures to the indexer exceeds 'maxFailuresPerInterval' in the specified 'secsInFailureInterval' seconds, the forwarder applies backoff. * The backoff time period range is 1-10 * 'autoLBFrequency'. * Default: 1 backoffOnFailure = <positive integer> * The number of seconds a forwarder backs off, or stops sending events, before attempting to make another connection with the indexer. * Default: 30 Can forwarder skip backoff? Backoff settings will be ignored by forwarder if following is set. autoLBFrequencyIntervalOnGroupFailure = <integer> * When the entire target group is not reachable, 'autoLBFrequencyIntervalOnGroupFailure' is the amount of time, in seconds, that a forwarder waits before attempting to connect to a target host in the group. * While 'autoLBFrequencyIntervalOnGroupFailure' is in effect, 'autoLBFrequency' is ignored. Once first connection is established to a group, 'autoLBFrequency' comes into effect again. * This setting is applied only when 'autoLBFrequencyIntervalOnGroupFailure' is less than 'autoLBFrequency'. * Every 'autoLBFrequencyIntervalOnGroupFailure' seconds, a new indexer is selected randomly from the list of indexers provided in the server setting of the target group stanza. * -1 means this setting is not active. * Default: -1    ... View more

Recently discovered that tcpout can read PQ (assuming PQ has lot of data due to indexing tier outage/Rolling restart etc.) at unbounded speed once indexing tier is up and connected, but can not push events at the same speed over TCP layer. This results in high memory spike and potentially OOM. If you have 12GB RAM and persistentQueueSize=18GB then you are likely to hit the issue of OOM if PQ is full or has more than 12GB data. However if you have 12GB RAM and persistentQueueSize=8GB or less then you are unlikely to hit OOM. Until fix patches are available,  set persistentQueueSize less than average available memory of the system. Issue is fixed for  upcoming 9.4.13/10.0.7/10.2.4/10.4.1 patches. ... View more