Hi everyone! Here are a few questions from the session, as well as the link to the on-demand recording (get the full Q&A deck and recording in the #office-hours Slack channel as well)  Q1: What does the transition path look like from reactive alerting to closed-loop, resilience-oriented feedback systems in AI-driven environments? A1: We are evolving from reactive OOTB and custom detector alerts to a human-in-the-loop model where AI agents use metrics, traces, and logs for automated troubleshooting. ​ ​ By applying security guardrails and granular permissions, agents learn safe remediation actions over time. This transition establishes the foundation for closed-loop, resilience-oriented feedback systems Q2: When AI agents initiate autonomous remediation, where are institutions anchoring the accountability surface - configuration, model approval, or runtime intervention? A2: When AI agents initiate autonomous remediation, institutions are increasingly anchoring the accountability surface in runtime intervention. While configuration and model approval provide the necessary foundation for safety, the most critical accountability anchor occurs when the troubleshooting agent presents its findings and proposed resolution to a human operator. By requiring human authentication before the remediation agent executes any changes, the institution ensures that the final decision-making authority remains with a human, effectively bridging the gap between autonomous execution and operational oversight. Q3: How does the MCP server differ from the Splunk Observability AI Assistant I’m already using? A3:The Splunk Observability's AI Assistant is a built-in chat interface designed for human users to interact with Splunk Observability data, accelerating troubleshooting directly in the Splunk Observability Cloud UI using natural language. ​ In contrast, the MCP server is a backend protocol that allows external AI agents (like Claude or custom agents) to programmatically access Splunk data and tools, enabling automated, cross-platform workflows that operate independently of the Splunk UI.​ ​Documentation: ​ ​Interact with your observability data using the Splunk MCP server Q4: It would be nice to see a demo on using Splunk to find shadow IT/OT and AI? A4:As part of Cisco AI Defense, AI Cloud Visibility automatically identifies AI assets within custom-built applications across your distributed environment, including unsanctioned workloads. This provides a centralized view, enabling you to inventory and assess your organization's AI security exposure.​ ​Coming soon, AI Agent Monitoring will be integrated with Cisco AI Defense to pinpoint AI risks and vulnerabilities. ​ ​Documentation: ​ ​Cisco AI Defense: AI Cloud Visibility ​ ​​ ​   ... View more

Hi everyone! Here are a few questions from the session (get the full Q&A deck and live recording in the #office-hours Slack channel)    Q1: What are the latest security content updates from the Splunk Threat Research Team? A: Security content releases: ESCU 5.19 & 5.20 delivered 11 new analytic stories and 55+ new detections, plus enhancements and tagging across 160+ existing analytics covering browser hijacking, cloud/Kubernetes threats, network devices, supply chain attacks, malware, and AI abuse.   Technical research & blogs: In the last quarter, we published 5+ deep-dive blogs and community updates covering emerging threats such as Castle RAT, PromptFlux / SesameOp, React2Shell RCE, LLM-driven malware concepts, and quarterly/monthly security content recaps.   Q2: What new analytic stories and detections are AI-enabled workflows that detect and respond to emerging threats across critical enterprise platforms? A: AI & GenAI abuse detection  New analytic stories for Shadow AI (local LLMs like Ollama) and Microsoft 365 Copilot to detect unauthorized AI usage, prompt injection, anomalous API activity, data/model exfiltration, and compromised identities.   Malware leveraging AI Expanded detections for LokiBot and new coverage for PromptLock, a GenAI-driven ransomware PoC, detecting malicious AI model usage, suspicious DNS activity, automated script generation, and encryption behavior.  AI-accelerated detection development Building an AI-assisted workflow to speed up creation, testing, and validation of detection YAMLs for faster response to emerging threats.   Q3. I'd like to know how to detect an active incursion or other C2C communications A:  Analytic stories around Shadow IT and Usage of AI -Remote Monitoring and Management Software -Scattered Lapsus$ Hunters -Suspicious User Agents -NetSupport RMM Tool Abuse     -Suspicious Local LLM Frameworks ... View more

Hi everyone! Here are a few questions from the session, as well as the link to the on-demand recording (get the full Q&A deck and recording in the #office-hours Slack channel as well)  Q1:  a) Can you show us some best in-class examples of executive dashboards made in ITSI?​ ​b) Interested in examples of Executive Dashboards for visualizing/comparing the health of a set of critical services. Answer​ Focus on Executive-Relevant Metrics​ High-level KPIs like service health scores, availability, and business impact indicators.​ Clear & Intuitive Visualizations​ Real-time data, color-coded health status, and clickable drill-downs.​ Show Service Relationships & Dependencies​ Visualize dependencies to understand impact across services.​ Provide Business Context​ Correlate IT health with business outcomes and customer experience.​ Keep Interface Simple & Focused​ Limit KPIs and services shown; use summary views with drill-down options.​ This approach empowers executives with a clear, actionable view of critical service health for informed decision-making and rapid response​. Documentation:​ Splunk Docs: Glass Tables​ The Top 10 Glass Table/Dashboard Design Principles to Boost Your Career and Your Business Q2: Can you give an overview of Event Analytics and demonstrate how Event IQ adds value here? Answer:​ Event Analytics (EA) is an alarm aggregation platform that is a companion tool to Service Analytics but may be used separately as a Monitor of Monitors​ Event analytics accepts alarm conditions ("notable events") from Service Analyzer as well as external (non-ITSI) tools:​ Splunk Alerts​ Splunk O11y Cloud​ Non-Splunk (3rd party) monitoring tools and solutions​ EA performs alarm deduplication and grouping into "episodes" (bundled related alarms) which can trigger any defined external action via an ITSM tool, Splunk alert actions, Splunk On-Call, messaging apps and tools, orchestration software...​ Event IQ is a new ML-powered feature in EA to perform intelligent alarm grouping into episodes with minimal-preconfiguration or understanding of alarm source relationships​ Currently: define key fields and their perceived values to be weighted in alarm grouping decisions​ Coming: full field discovery for grouping/weighting purposes​ Documentation:​ Event Analytics: https://docs.splunk.com/Documentation/ITSI/4.20.1/EA/AboutEA ​ Event IQ: https://help.splunk.com/en/splunk-it-service-intelligence/splunk-it-service-intelligence/detect-and-act-on-notable-events/4.21/event-correlation/automate-event-correlation-with-event-iq-in-itsi ​Q3: How do you create a KPI without using a content pack, such as AppDynamics?​ Answer​ KPI creation and configuration is done via the Service configuration UI as well as the Service Template configuration UI. High level config steps:​ Step​ Task​ Description​ Optional/Required​ 1​ Define a KPI source search​ A search string that you define as the basis for your KPI, using a data model, an ad hoc search, a metrics search, or a base search.​ Required​ 2​ Split and filter by entities​ Break down the KPI to apply the search to multiple entities, enabling comparative analysis of search results on a per-entity basis. Filter entities in or out of the KPI search.​ Optional​ 3​ Configure KPI monitoring calculations​ The recurring KPI search schedule and the statistical operations performed on the search results, including service health score calculations.​ Required​ 4​ Define KPI unit and monitoring lag​ Define the unit of measurement to display for the KPI. Configure the monitoring lag to offset indexing lag.​ Optional​ 5​ Enable backfill​ Fills the summary index with historical raw service health score data.​ Optional​ 6​ Configure KPI thresholds​ Severity-level thresholds that you apply to KPI search results. Thresholds let you monitor KPI status (normal, low, medium, high, and critical) and set trigger conditions for alerts.​ Required​ 7​ Configure KPI thresholds with machine learning in ITSI​ Use machine learning to analyze your KPIs with existing data and generate recommendations for optimal threshold values. Thresholds let you monitor KPI status (normal, low, medium, high, and critical) and set trigger conditions for alerts.​ Optional​ ​ Documentation:​ Splunk Docs: Create KPIs​ Splunk Lantern: Using SRE golden signals for KPIs Question 4: What is “drift detection” and how is that used?​ Answer:​ ​ Drift detection is used to identify changes in KPI behavior that happen over a long period of time not normally captured by Adaptive Thresholding – the "frog in the boiling pot of water" scenario​ Adaptive Thresholds: training max 60 days​ Drift Detection: training min 90 days   ArifV_0-1764792538184.jpg   ArifV_1-1764792547962.jpg Documentation: Splunk Docs: Configure Drift Detection​ Splunk Docs: Migrate Anomaly Detection to Adaptive Thresholding​ Q5: What are best practices to deal with events as fast as they are generated? Answer​ Identify the alert data sources​ If ITSI supports alerts data integration for these sources (i.e., tools), use those ootb onboarding and normalization templates to normalize alerts​ Use lookup-based enrichment policies for enriching alerts from sources such as CMDB​ Use ootb NEAPs provided in Content Pack for monitoring and alerting for aggregating / grouping alerts ​ Be on ITSI 4.21 version to make sure of these latest capabilities incl. New queue-based event pipeline that brings higher scale and reduced latency ​ Documentation:​ https://help.splunk.com/en/splunk-it-service-intelligence/splunk-it-service-intelligence/detect-and-act-on-notable-events/4.21/overview/overview-of-event-analytics-in-itsi ​ Question 6: Can we learn more about ITSI integrations with Teams and Slack?​ ​ Answer​ Splunk ITSI’s Teams and Slack integrations are Splunk-supported & maintained, available on the Splunkbase website.  The integrations (content packs) do way more than help notify teams: ​ They streamline automatic alerting for correlated events and service health notifications They support actions and workflows within Teams and Slack, like posting messages, getting info about users, or even starting a SlackBot and making health checks In Teams, configure potential actions that allow for 3rd-party interactions between Splunk or a 3rd-party app from Teams ​ Documentation:​ Gain quick access w/ the Splunk App for Content Packs ​ Splunkbase apps for Slack, and Teams​ (Check here for troubleshooting Teams)​ ​ ... View more

You can view the On-Demand recording here (pw: FpvBfrn6) and slide deck here, but here is a sampling of questions covered in the event. Q1: Can I get some guidance on Agent installation and configuration? A: For AppDynamics, there are many ways to install and configure an Agent.  Below are the options: Manual configuration and installation Follow the getting started wizard on the Overview page Using automation like ansible or some other orchestration system to deploy applications and inject the agent and configuration If containerized and running in Kubernetes, you could use the auto instrumentation feature of the AppD cluster agent to deploy agents AppDynamics has a feature called SmartAgent that can install, configure and manage agents once smartagent is installed. Documentation: Getting started with application Agents Using Automation to install and manage application Agents Use auto instrumentation for Kubernetes using the AppD Cluster Agent Using SmartAgent to install and manage agents Q2: What is the best way to identify failure/error trends on a periodic basis (ex: after a software release of our internal product)? A:  Define the time frame you want to compare, where one is the older application code, and the second is the new release Create custom Dashboards and Health Rules displaying key error metrics (e.g., Error Rates, Response time, calls per minute) for your product’s  overall application performance, tiers and business transactions. Dash Studio has a built in feature that allows you to timeline comparison using key metrics and business transactions using Widget time ranges Documentation: Dash Studio Widget Time Ranges Q3: What are some best practices for filtering out “white noise”? A: "White noise" in AppDynamics typically refers to excessive alerting especially one that are not actionable. Optimize Error Configuration and Slow Transaction Thresholds  For Health Rules combine multiple conditions to get a clear technical problem definition for each monitor event (e.g., "Error Rate > 5% AND Calls Per Minute > 100")  Fine-Tune Health Rules making sure we set conditions with realistic thresholds and proper baselines  Use the <n> times in a time period to avoid single outliers creating monitor events.   Optimize Business Transaction (BT) Detection and Configuration, exclude irrelevant BTs those that are not critical to your business or application performance, lock down the relevant BT and review on a regular basis.  Don’t overdo it focus on alerts that actionable Take Advantage of Anomaly detection  Documentation: Dynamic Baselines Setup Health Rules Refine Business Transactions Anomaly Detection Q4: Does AppDynamics support OpenTelemetry? A:  Yes, AppDynamics SaaS currently supports OpenTelemetry Traces and Spans AppDynamics On-Premise Virtual Appliance is planned to support OTEL in the near future   AppDynamics also provides its own version of the OTEL collector that can be configured to send data to the controller OpenTelemetry end point In Americas, its supported in Oregon and Sao Paulo Regions In EMEA, its supported in Frankfurt and coming soon to London In APAC, its supported in Mumbai, Singapore and Sydney Documentation: AppDynamics OpenTelemetry Support Supported Regions Q5: Can you share some guidance around JVM Tuning? A: Appdynamics recommends that in resource-intensive environments where Splunk AppDynamics features, such as asynchronous transaction tracking, developer mode that result in a high number of metrics or snapshots reported per minute can affect agent resource consumption. In these cases, its recommended to increase the Heap and PermGen settings for the agent. Maximum heap size (-Xmx): 100 MB in addition to the amount required by the application Maximum PermGen (permanent generation) heap size (-XX:MaxPermSize): 20 MB in addition to the amount required by the application. The setting for MaxPermSize is applicable only for Java 6 or Java 7.  JVM option is no longer relevant in Java 8 and later versions Documentation: Java agent resource overhead Tune Java Agent Performance ... View more

You can view the On-Demand recording at the link above and slide deck here, but here is a sampling of questions covered in the event. Q1: Can you help me understand the two different types of anomaly detection in ITSI?  A: Anomaly Detection within Splunk ITSI uses machine learning to detect trend & event level anomalies, and alert teams. ITSI analyzes when data deviates from expected patterns, thresholds, and historical behavior to provide insights into detected patterns & uncover new patterns across multiple events that could indicate a potential issue. Entity cohesion anomaly detection analyzes asset data in a peer group, and identifies when some deviate from their common behavior Trending anomaly detection analyzes patterns in events for both sudden changes, and long-trending historical anomalies Documentation:  Understanding anomaly detection in ITSI Apply anomaly detection to a KPI in ITSI Anomaly detection algorithms Q2: What is adaptive thresholding and why is it important? A: In Splunk ITSI, adaptive thresholding is a feature that uses machine learning to dynamically adjust alert thresholds based on historical data patterns, rather than relying on static, unchanging values. This allows for more accurate anomaly detection in environments where data behavior fluctuates over time. By analyzing trends and normal variations, adaptive thresholds reduce false positives and missed alerts, ensuring that operational teams are notified of genuine issues even as the baseline behavior changes. Reduced false positives Improved anomaly detection  More accurate alerting Time-saving Documentation:  Create Adaptive Thresholds in ITSI Knowing proper adaptive threshold configurations Maintaining adaptive thresholds Q3: How do I configure and enable the Notable Event Aggregation Policy? (NEAP) A: What is a NEAP?  Notable event aggregation policy is the fundamental unit of event grouping in ITSI. NEAPs are the data structure the Rules Engine uses to group notable events into deduplicated episodes and organize them in Episode Review.  These episodes have their own title, description, severity, status, and assignee that are separate from the individual notable events within the episode. Aggregation policies are also the container for action rules that automate episode actions, such as sending an email or pinging a host. How to configure & enable NEAPs? Configuration > Event Management > Notable Event Aggregation Policies > Create NEAP Ships with default NEAP -> default policy only receives notable events matching no other aggregation policy's filtering criteria Content Pack for Monitoring & Alerting  Documentation:  Aggregation policies in the Content Pack for ITSI Monitoring and Alerting Best practices for implementing Event Analytics in ITSI Q4: What are some tips to reduce alert noise? A:  Optimize alert grouping with Event Aggregation Improve Alert Hygiene (get your thresholds right) Enrich Entity metadata Ensure your KPIs are relevant and meaningful Leverage the Content Pack for Monitoring and Alerting contains tools to analyze your episodes and alerts Documentation: https://www.splunk.com/en_us/solutions/alert-noise-reduction.html https://lantern.splunk.com/Observability/UCE/Guided_Insights/Reduce_Noise https://docs.splunk.com/Documentation/CPITSIMonitorAlert/2.3.0/CP/About Q5: Can we learn more about ITSI integrations with Teams and Slack? A: Splunk ITSI’s Teams and Slack integrations are Splunk-supported & maintained, available on the Splunkbase website. The integrations (content packs) do way more than help notify teams:  They streamline automatic alerting for correlated events and service health notifications They support actions and workflows within Teams and Slack, like posting messages, getting info about users, or even starting a SlackBot and making health checks In Teams, configure potential actions that allow for 3rd-party interactions between Splunk or a 3rd-party app from Teams Documentation: Gain quick access w/ the Splunk App for Content Packs  Splunkbase apps for Slack, and Teams (Check here for troubleshooting Teams) ... View more

Hello! Here are the recap materials from the session: On-Demand recording Slide Deck Here are the questions that we received in the session (more detailed solutions and info can be found in the slide deck) Q1: What is the best strategy to reduce ingestion costs? is there a way to show how much is being reduced?  A:  Archive all Unused Metrics Figure out lowly used metrics Sign up for Automated Archiving to Automate Documentation: Pipeline Management  Usage Analytics Data Management More Usage Story(Blog) Archived Metrics(Blog) Logs Metricization  The New Rules of Data Management (E-Book) Building a self service observability practice (Tech Talk) Q2: What are the best recommendations relating to data retention in Splunk, or even cold restore? A:  Metrics 1 Second Resolution: 3 Months 10 Seconds or more: 13 Months APM Raw Traces: 8 Days Traces of interest viewed in the Splunk APM user interface: up to 13 Months Profiling Data: 8 Days RUM Spans: 8 Days Session Replay: 8 Days Synthetics Run Results: 8 Days Metric Data: 13 Months Logs Varies by industry, regulations, source, and organization policies. See the Splunk Lantern article for more details and recommendations. Other Troubleshooting MetricSets: 8 days Monitoring MetricSets: 13 Months Documentation: Data Retention in Splunk Observability Cloud Configure Extended Trace Retention Setting data retention rules in Splunk Cloud Platform Learn about MetricSets in APM Q3: How would I manage log ingestion made by fluent like we do with Splunk Universal Forwarder and props? A: The Fluent Forward receiver allows the Splunk Distribution of the OpenTelemetry Collector to collect events using the bundled Fluentd application. The receiver accepts data formatted as Fluent Forward events through a TCP connection. All three Fluent event types, message, forward, and packed forward, are supported, including compressed packed forward.  However, this integration will be deprecated in October of 2025 as we have shifted to native OpenTelemetry log collection. Our best practice recommendation is as follows: Kubernetes: OpenTelemetry native log collection Stand-alone Linux/Windows hosts: Universal Forwarder log collection When used with Splunk Log Observer Connect, you can take advantage of effectively all Splunk Observability Cloud logging functionality, including Related Content. Documentation: Fluent Forward Receiver Splunk Distribution of the OpenTelemetry Collector Log Observer Connect Related Content ... View more

Hello! Here are the recap materials from the session: - On-Demand Recording - Slide Deck And here are the questions we received (there are supplemental slides for each of these solutions in the deck) Q1: What are some unique advantages of Splunk IM? Real time, OTel Native, AI-Assisted, End-to-End Correlation, all at Enterprise Scale (see additional slides in deck) Documentation: https://www.splunk.com/en_us/products/infrastructure-monitoring-features.html Q2: What is related content? A: The Related Content panel in Splunk Observability Cloud automatically correlates and presents data between different views within Splunk Observability Cloud (see add'l slides in deck) Documentation: https://help.splunk.com/en/splunk-observability-cloud/data-tools/related-content Q3: How can I send metrics to Splunk IM? A: With OpenTelemetry Collector, Cloud APIs, REST APIs (see add'l slides in deck) Documentation: https://help.splunk.com/en/splunk-observability-cloud/monitor-infrastructure/set-up-infrastructure-monitoring Q4: How to manage log ingestion made by fluent, like we do with Splunk universal forwarder and props?  A: You can deploy an Otel Collector with a fluentforward receiver and export to Splunk HEC endpoint Documentation: https://github.com/signalfx/splunk-otel-collector/blob/main/cmd/otelcol/config/collector/agent_config.yaml https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/splunkhecexporter Q5: Is IM only for Kubernetes? What if I want to monitor servers and apps hosted in AWS via IM? A: With Splunk IM you can get metrics from hosts and containers on-prem or in the cloud (AWS, Azure, GCP) and services running on them like K8s, message brokers, web servers, DBs etc You can send infrastructure metrics to Splunk IM:  With OpenTelemetry Collector (deployed in EC2 instances) By calling Cloud APIs to get metrics from AWS (CloudWatch metrics) By using REST API calls to send custom metrics Documentation https://help.splunk.com/en/splunk-observability-cloud/manage-data/available-data-sources/supported-integrations-in-splunk-observability-cloud#nav-Supported-integrations ... View more

Thank you to all who attended this event! For those who weren't able to attend, here are the recap materials: - Q&A Slide Deck - Live Recording Please reply here if you have any questions! ... View more

Here are some of the questions covered in the session: Q1: How do you get logs from a Kubernetes Cluster? Solution:  Set Log Collection in the wizard: ArifV_0-1734031777571.png     See documentation - covers cases like host logs, multi-line logs, using pod annotations, sending events, etc. There are also many more tips about this topic from slides 8-14 in this deck Documentation: https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-kubernetes/kubernetes-config-logs.html Q2: How can I optimize troubleshooting for K8s alerts? Solution:  Utilize the Navigator links embedded in alerts  Autodetector alerts Utilize built-in metrics and dashboards Identify problematic nodes, pods, and containers using the hierarchy map in the Kubernetes Navigator Deploy the Splunk Distribution of the Otel Collector to your cluster for correlation of metrics, traces and logs (related content) Enrich telemetry data with custom metrics or adding relevant metadata to enhance troubleshooting  Documentation: https://docs.splunk.com/observability/en/infrastructure/monitor/k8s-nav.html https://docs.splunk.com/observability/en/gdi/get-data-in/compute/k8s.html#get-started-k8s https://docs.splunk.com/observability/en/gdi/opentelemetry/collector-how-to.html#collector-how-to  https://docs.splunk.com/observability/en/metrics-and-metadata/relatedcontent.html Q3: One of my containers produces events as json. Any pointers on how to teach the OTel Collector to read it in as json, I am getting multiple events strung together as one. Ideally is this something that I can achieve in the Splunk Otel helm chart? Solution:  See details below on processing multi-line logs; this is likely why you are getting a multi-line json showing up as individual log lines Documentation: https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md#processing-multi-line-logs  Installing with YAML manifests ... View more