Metric streaming, a method that employs Kinesis Data Firehose Stream for the delivery of metrics, is an advanced alternative to traditional metric polling, which may exhibit a latency of 5-10 minutes. This highly scalable and efficient approach ensures that, once set up, near real-time metrics start flowing in just 1-2 minutes.
We've enhanced the ways in which you can set up metric streams directed to Splunk Observability Cloud. Previously, enabling metric streams was limited to our API and the setup required invoking CloudFormation templates which create required infrastructure on AWS and granting Cloudwatch stream permissions that could let us create and manage metric streams for you. As those streams’ lifecycles would be bound to Splunk integrations, they are now called "Splunk-managed streaming". We've now incorporated the Splunk-managed streaming option within the UI guided setup alongside the polling method. Concurrently, with the introduction of Quick AWS Partner Setup in the CloudWatch console, we've integrated an AWS-managed streaming capability, enabling users to efficiently manage metric streams via Amazon CloudWatch. You can find how to set up the AWS-managed metric streams here. A detailed comparison of these options is also available in our documentation.
If you have previously enabled the metric streaming option via API (which will now be called “Splunk-managed streaming”), rest assured that your integrations will continue to function as usual. A possible alteration you may observe relates to those specific integrations where an access token has been assigned to Kinesis Firehose via AWS CloudFormation templates while no token or a different one was assigned to the integration in Splunk Console, you might notice a shift in the token usage metrics. This is due to our transition towards exclusively utilizing the token configured on the AWS side. Consequently, this also streamlines the guided setup process for AWS integrations with streaming options, eliminating the need to select an access token. For current insights into which metric streams on AWS and integrations on Splunk exhibit token discrepancies, you could create a new chart for the metric “sf.org.awsMetricStreamsTokenDifference” and get all the details in the “Data table” view.
Depending on your chosen streaming method, filtering metric streams can be executed using namespaces and metric names, either within Splunk Observability Cloud or the AWS CloudWatch console. It's important to note that, when transitioning from a polling setup with existing filters, some filters may not be compatible with streaming configurations. Specifically, functionalities such as resource tag filtering and advanced filtering mechanisms are not supported in metric streaming integrations.
We have also expanded coverage for the Cloud Metric Metadata Sync service that collects resource group tags and entity properties from AWS APIs. This expansion mainly targets the metrics from Amazon Keyspaces with the namespace “AWS/Cassandra”, which offers a convenient and scalable way to run Cassandra databases. The corresponding metadata items are automatically collected as property metadata for all new and existing metrics related to this namespace. To ensure all of your metadata is collected, please verify that your AWS IAM permissions for the integration include all prescribed permissions from here.
Additionally, our dedication to user-centric improvements continues with detailed status visibility options for AWS integrations. You can now view specific states for Optimizer, Metric Polling, Metric Streaming, and Log Streaming (tailored for Log Observer customers). You can find it straightforward in the UI to pinpoint the exact state of integrations, making it easier to identify and address issues like stopping and cleaning streaming.
We are thrilled to announce that this release will first be available in selected realms, ensuring a smooth rollout. Following this initial phase, we will offer general availability to all our users on November 6, 2023.
