Splunk Observability Cloud

Too many timeseries, filter on values

ReneVisser
New Member

We have create a lot of custom metrics based on sensu monitoring. These metrics can have a value of either 0 (ok), 1 (warning), or 2 (critical)
I want to build a chart base on a query like 
data('sensu_*', filter=filter('sf_environment', 'prd') and (filter('owner', 'teamA') or filter('type','platform'))

this query returns over 3000 timeseries so results are sampled:
(Rendering 1000 out of 3270 time series. Values displayed may not be representative; consider filtering your chart or applying aggregation analytics.)

In the end I want to have something like this:
A = data('sensu_*', filter=filter('sf_environment', 'prd') and (filter('owner', 'teamA') or filter('type','platform')), rollup='max').between(1,3).count(by='host.name').above(0).publish(label='Critical', enable=True)

but now the result from this query isn't trustworthy.


There no extra dimensions to filter upon, but I want to filter all the results where value is 0.
when I use methods like above(), between or when i create a new stream with when() I keep getting this warning.
Does anyone have a suggestion what I can try.

Labels (1)
0 Karma

bishida
Splunk Employee
Splunk Employee

Hi,
The wildcard * in the metric name may be causing multiple MTS to be returned and all being represented as "sensu_*". Maybe try specifying a single metric name and see if that helps. If you have custom metrics in addition to other sensu metrics, you may need to create a way to filter just on those custom metrics (e.g., alter the name like sensu_mycustom_* or add a dimension that indicates it's custom)

0 Karma

ReneVisser
New Member

Thank you for the suggestion,
But we need all the sensu checks to be monitord, so we need all the metrics.
I did however find a dimension which helped me out, coincidentally there was an output dimension which contains "Ok" for all the datapoints containing a 0 as a value for all the metrics. So I was able to. use that.

But still I am wondering if it somehow is possible to filter an the value of metrics.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...