Splunk Search

How would I build a table of all metrics and their dimensions in splunk

Explorer

My splunk server is receiving metrics from collectd.

I want to build a table showing the metrics, dimensions, and values emitted for each unique metric_name/dimension/host combination.

| metirc_name | dimension_1 | ... | dimension_n | avg_value over 10s | <- it would look something like that

Ideally what I'm asking for is something like this:

| mstats WHERE source=*, metric_name=*, index=* BY *  | table *
  • I want the value for each unique combination of metric_name and dimension and value for a given. But wildcards are not allowed in the BY clause.
  • I need to fill in empty dimensions because not all values will have the same dimensions.

I don't see a way to query with out explicitly defining each dimension key. Additionally I don't see a way to extract the raw datapoints from splunk. I could settle for averaging over a time span for the unique combination of host, metric_name, and all dimensions.

Any help is greatly appreciated!

Builder

try :

| mcatalog values(_dims) WHERE "index"="*" GROUPBY metric_name index host
| rename values(_dims) AS dimensions 
| table metric_name dimensions index host

Esteemed Legend

You need some combination of this:

| mcatalog values(_dims) WHERE index="*" AND sourcetype="*" BY index metric_name

And this:

| mstats latest(_value) WHERE index="*" AND sourcetype="*" AND metric_name="*" BY metadata
0 Karma

Explorer

Ask the similar question ...

In Prometheus, there is such a concept called "all time series at current time", how can we achieve the same in Splunk?

0 Karma

Explorer

I'm having trouble figuring out how to access the dimensions and dimension values of metrics also. Plenty of documentation on how to list out the dimensions, but none for their values.

0 Karma

Splunk Employee
Splunk Employee

I don't think there is a way to get all of the data out of a metrics index like this, what are you trying to achieve (why are you trying to do do this)?

If you have just 100 metrics, each with 5 dimensions, each with just 10 values that'd still be a table with 5,000 rows - that's more information than is appropriate to show a user in a table.

To list the dimensions and their values you use the mcatalog command:

| mcatalog values(_dims) WHERE metric_name=* AND index=*

| mcatalog values(your_dim) WHERE metric_name=* AND index=*
0 Karma

Splunk Employee
Splunk Employee

"I could settle for averaging over a time span for the unique combination of host, metric_name, and all dimensions."

| mstats avg(_value) where index=* AND metric_name=* earliest=-1h latest=now by host, metric_name
0 Karma

Explorer

Thanks for replying, but this does not get me the unique values for host, metricname, and all dimensions. It only gives me unique values by host and metric name.

The problem I'm facing is that I want to group by each dimension, host, and metric name with out explicitly defining each dimension in the query.

0 Karma