Many thanks ... View more

@livehybrid  thanks for your reply but i am confusing with this: By default, Splunk software creates each data model acceleration summary on the indexer, parallel to the bucket or buckets that cover the range of time over which the summary spans, whether the buckets that fall within that range are hot, warm, or cold. If a bucket within the summary range moves to frozen status, Splunk software removes the summary information that corresponds with the bucket when it deletes or archives the data within the bucket. Look at my senario bellow of understanding is correct or no : Scenario: Volume 1 (Hot/Warm/DMA) and Volume 2 (Cold) If you have configured your index to use Volume 1 for homePath and summaryHomePath, and Volume 2 for coldPath, here is how the data flows: 1. The Directory Structure The indexer will split the data across your mount points like this: Volume 1 (/mnt/fast_disk/): Hot/Warm Buckets: Stores the rawdata and the datamodel_summary together. Standalone Summary Directory: If a bucket moves to Cold (Volume 2), but your configuration tells Splunk to keep summaries on Volume 1, Splunk creates a mirrored directory structure on Volume 1 just to hold the .tsidx files. Volume 2 (/mnt/cheap_disk/): Cold Buckets: Stores only the rawdata (and standard index files like bloom filters). 2. What happens when data "Rolls"? From Warm to Cold When a bucket reaches the age or size limit to move to Cold: The Raw Data moves from Volume 1 to Volume 2. The DMA Summary behavior depends on your config: Default: The summary moves with the bucket to Volume 2 (the cheap disk). Optimized: If you set summaryHomePath specifically to Volume 1, the raw data moves to Volume 2, but the summary stays on Volume 1. This is a "Best Practice" because it keeps your accelerated searches running on your fastest storage even for older data. From Cold to Frozen When the bucket rolls to Frozen: Original Data: Moved to your archive or deleted from Volume 2. DMA Data: The software looks at wherever the summary was stored (Volume 1 or Volume 2) and deletes it immediately     ... View more

  @livehybrid  thanks for topic sharing  but the bellow sentence confuse me :   Although data model acceleration summaries are unbounded in size by default, they are tied to raw data in your index buckets and age along with it. When summarized events pass out of cold buckets into frozen buckets, those events are removed from the related summaries. is what I understand in the example below (adapted from GPT) correct? 📁 1. Raw Data (The Parent Bucket) Location: $SPLUNK_HOME/var/lib/splunk/net_logs/db/       Example Structure:     db_1715001600_1715000000_101/ ├── rawdata/ │ └── journal.gz # Contains the actual raw log data ├── 1715001600-1715000000-123456789.tsidx # Raw index file └── Strings.xml # Metadata for string lookups       🧠 2. Data Model Acceleration (The Shadow Copy) Location:     $SPLUNK_HOME/var/lib/splunk/net_logs/datamodel_summary/       Structure Explanation: Splunk creates a folder named after the Data Model ID Inside it, bucket directories mirror the original index buckets Example Structure:     DM_Project_Security_Network_Traffic/ └── db_1715001600_1715000000_101/ # Matches the original bucket ID ├── 123456789.tsidx # Accelerated (optimized) data └── hash_info.xml # Metadata for acceleration     🔄3. Lifecycle Demonstration (Step-by-Step) Step 1: Ingestion & Acceleration You ingest  firewall logs Splunk creates a bucket:   bucket_101   Since Data Model Acceleration is enabled: The Background Summarization Processor scans this bucket It generates a smaller, optimized .tsidx file This file is stored in:   datamodel_summary/DM_Project.../db_..._101/   Step 2: Aging & Deletion (Retention Policy Enforcement) Your retention policy is set to 30 days Once bucket_101 becomes 31 days old: Splunk marks it as expired What happens next: Bucket Removal Splunk deletes the raw data bucket:   $SPLUNK_HOME/var/lib/splunk/net_logs/db/db_..._101/   Automatic Cleanup of Accelerated Data Splunk checks the corresponding Data Model Summary path It finds the matching bucket:   datamodel_summary/.../db_..._101/   This directory is automatically deleted ✅Key Takeaways Data Model Acceleration creates a dependent “shadow copy” of indexed data Accelerated data is tightly coupled to its original bucket When the original bucket is deleted (due to retention policies), the accelerated summary is also removed automatically   ... View more

According to SVA --> "Using a network load-balancer (L4/L7) to provide a highly available syslog collection infrastructure is discouraged due to potential data loss and fidelity concerns (IP lost in proxy). The recommended approach is to use a cluster resource manager solution (e.g. pacemaker/keepalived) or a VM-based syslog server deployment instead. " ,    did any one try this ? ... View more

is this best practice or  mandate --> ES doesn't play well with other apps and so needs to be on its own. ... View more