Hi,  I am trying to better understand how to correctly implement incremental backups in Splunk, and I would appreciate clarification on a few points regarding bucket structure and disaster recovery. We have a single AIO instance. 1. Bucket Structure and Incremental Backups When data rolls to the frozen state, I observe that the bucket contains only a single file (journal.zst). However, in warm buckets, the rawdata directory contains multiple files such as: journal.zst, slicemin.dat, slicesv2.dat Q1: What is the role of these files within the bucket structure? Specifically, how do slicemin.dat and slicesv2.dat relate to journal.zst? Q2: If I want to implement incremental backups, is it sufficient to back up only journal.zst, or is it necessary to archive the entire bucket (including all associated metadata and index files)? 2. Disaster Recovery and License Considerations In a disaster recovery scenario (e.g., disk failure), assume the following steps: - A new VM is provisioned - Splunk is installed and configured (including restoring configuration and license data) - Splunk recreates the necessary directory structure At this point, I have archived copies of warm and cold buckets (Based on the answer to Q1 and Q2 its a full bucket/directory or just the journal.zst). Q3: Where should archived warm and cold buckets be placed within the indexer’s directory structure to ensure proper recognition? Q4: After restoring these buckets, does Splunk need to re-index the data, or will the data immediately be searchable since it was already indexed prior to backup? Thank you in advance for your guidance. ... View more

Hi there, We’re seeing consistent ingestion failures with the Message Trace (MT) input in the Splunk Add-on for Microsoft Office 365. - Authentication to Microsoft 365 succeeds. - However, every request to the Message Trace endpoint returns HTTP 500, regardless of how small the query window is. - To rule out an overly large time range, i cloned the MT input and tested with a very small window (e.g., 15 minutes for today’s data). The request still fails with repeated 500 responses. - Other inputs are OK. 2026-03-05 13:12:36,078 level=ERROR logger=splunk_ta_o365.modinputs.message_trace datainput="MT_test1" start_time=1772712742 message="HTTP Request error: HTTPSConnectionPool(host='reports.office365.com', port=443): Max retries exceeded with url: /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate eq datetime'2026-03-05T00:00:00Z' and EndDate eq datetime'2026-03-05T00:15:00Z' (Caused by ResponseError('too many 500 error responses'))" ... (stack trace omitted)  Q1: Have you seen this behavior (persistent HTTP 500) from the Message Trace Reporting Webservice endpoint? Q2: Are there known service-side limitations or tenant-specific issues that can cause this? Q3: What are the recommended next troubleshooting steps and/or mitigations? Q4: I know its a legacy MT endpoint as of march 2026 but there is no update from o365 addon and im not feeling to make an input for MT from scratch. Any info when the new update is planned? Thanks in advance.   ... View more