Supercharging Windows Security Detection Performance: Introducing Hybrid Field Extractions

[hrawat]

Windows event logs—from Security auditing and Sysmon to PowerShell script blocks—form the operational backbone of modern Security Operations Centers (SOCs). However, in high-throughput environments, running hundreds of threat detections against raw XML event streams has historically come at a steep compute cost.

Starting with the release of the [Splunk Add-on for Microsoft Windows 11.0.3]  (https://splunkbase.splunk.com/app/742), we are introducing native Index-Time Auto-Extraction (xmlkv-winevt) and intelligent Hybrid Extraction. With just a single configuration toggle, security teams can dramatically accelerate search performance and unlock greater detection concurrency across Enterprise Security (ES) and ESCU content. Requires Splunk 10.4.3/10.6 and above on Splunk Platform side.

---

The Challenge: The Search-Time REGEX Bottleneck

In a traditional setup, Windows XML events are ingested as raw text and parsed at search time:

• Over 350+ Enterprise Security Content Update (ESCU) detections rely on search-time key-value extractions.

• The Splunk_TA_Windows app evaluates over 120 regular expressions per matching security event at search time (with 500+ REGEXes defined across the TA).

• Scanning wide time ranges or high-volume event streams forces search heads to spend significant CPU and memory repeatedly parsing raw XML text.

This overhead introduces search latency, increases cluster compute pressure, and limits the number of real-time or scheduled detections an organization can reliably run.

Traditional Search-Time Flow: [ Ingest Raw XML ] ──> [ Search Dispatched ] ──> [ 120+ Regex Parsers per Event ] ──> [ High CPU / Latency ]

---

The Solution: Index-Time Auto-Extraction & Hybrid Processing

Splunk Add-on for Microsoft Windows 11.0.3 introduces native index-time extraction (xmlkv-winevt) tailored specifically for Windows XML event structures (e.g., EventID, Provider, EventData key-value pairs).

Instead of relying on regex execution during search queries:

1. Fields are pre-tokenized and indexed during ingestion.

2. Search filters hit indexed tokens directly (tsidx), bypassing the need to parse raw text with regex.

3. Smart Hybrid Optimization (OPTIMIZE_IE_EXTRACT) Splunk dynamically inspects search requirements. If all fields needed by a search are already indexed, expensive search-time regex extractions are automatically skipped.

Hybrid Index-Time Flow: [ Ingest & Auto-Extract ] ──> [ Fields Pre-Indexed ] ──> [ Search Dispatched ] ──> [ Fast tsidx Match & Zero Regex ]

---

100% Backward Compatible

A primary design requirement for this feature was seamless enterprise adoption without breaking existing data pipelines or detection logic:

• Historical Events: Events indexed prior to enabling index-time extractions continue to work seamlessly via standard search-time fallback regexes.

• Mixed Data Streams: If a detection query requests a custom or long-tail field that was not extracted at index time, Splunk automatically falls back to search-time extraction for that specific field.

• Zero Query Rewrites: Existing detection rules, macros, and datamodels require zero changes.

---

Getting Started: Enabling Index-Time Extraction

For Splunk Enterprise Security customers, the required configurations are bundled directly inside the [Splunk Add-on for Microsoft Windows (v11.0.3+)] (https://splunkbase.splunk.com/app/742).

1. Upgrade Prerequisites

• Splunk Platform: Version 10.4.3/10.6 or later (Splunk Enterprise or Splunk Cloud Platform).

• Technology Add-on: Install or upgrade to [Splunk Add-on for Microsoft Windows 11.0.3] (https://splunkbase.splunk.com/app/742) on your Search Heads and indexers( indexers need props.conf and transforms.conf for index time config).

2. The One Configuration Change

The extraction mode definition (INDEXED_EXTRACTIONS = xmlkv-winevt) is pre-packaged in the TA. To enable hybrid extraction, activate the typing pipeline in your local props.conf:

[source::XmlWinEventLog:*]
XML_INDEXED_EXTRACTIONS_PIPELINE = typing

---

Advanced Ingest Tuning & Options ( optional)

Splunk 10.6 provides granular controls in props.conf to balance tsidx storage size and search performance according to the official Splunk documentation:

---

Observed Performance & Operational Impact

Metric	Search-Time Extraction	Index-Time Hybrid (Splunk 10.4.3/10.6 + TA 11.0.3)
Search Latency / Run Time	High (scales with event volume)	Significantly Reduced (hits tsidx directly)
Search Head CPU & Memory	High (120+ REGEX evaluations/event)	Minimal (Search-time REGEX skipped)
Detection Concurrency	Constrained by CPU pressure	Higher detection throughput across ESCU
Storage Impact	Baseline	Modest increase in tsidx size
Backward Compatibility	Standard	Full seamless fallback for historical data

---

Next Steps

1. Review the props.conf documentation bundled inside the [Splunk Add-on for Microsoft Windows 11.0.3] (https://splunkbase.splunk.com/app/742).

2. Test the configuration in a staging environment by setting XML_INDEXED_EXTRACTIONS_PIPELINE = typing and comparing search run times on identical ESCU detection searches.

3. Deploy to production indexers/forwarders to scale your Windows security detection operations.

4. Retire Brittle Pre-Ingest Conversions: If your organization relies on custom/hacky workarounds to convert XML into JSON before sending to Splunk, you risk losing critical security keys or having events rejected by downstream indexer policies. Splunk’s out-of-the-box (OOTB) hybrid solution natively knows when an event or field is not eligible for index-time extraction and gracefully falls back to search-time extraction—ensuring zero data loss and complete event fidelity.