Hi, I started using tags by tagging my hosts with the environment they are in and the service the host. Using these tags in log/event indices works perfectly well, but I am not able to filter by tags in mstats. I tried many variations of "WHERE tag=env.prod" or "WHERE "tag::host"="env.prod" but none return any results. I checked that these tags really are there with mpreview which shows all the tags on the specific hosts and I also was able to filter with a small workaround using the tags command:   | mstats rate(os.unix.nmon.storage.diskread) AS read rate(os.unix.nmon.storage.diskwrite) AS write WHERE `my-metric-indizes` AND (host=*) BY host span=5m | tags | WHERE "service.vault" IN (tag) AND "env.prod" in (tag) | stats sum(read) AS read, sum(write) AS write by _time,host | timechart max(read) as read, max(write) as write bins=1000 by host   Is there a way to filter by a tag directly in mstats? The workaround is not very performance friendly... ... View more

Hi, I am trying to collect metrics from various sources with the OTel Collector and send them to our Splunk Enterprise instance via a HEC. Collecting and sending the metrics via OTel seems to work quite fine and I was quickly able to see metrics in my splunk index. However, what I am completely missing are the labels of those prometheus metrics in Splunk. Here an example of some of the metrics I scrape:   # HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter. # TYPE jmx_exporter_build_info gauge jmx_exporter_build_info{version="0.20.0",name="jmx_prometheus_javaagent",} 1.0 # HELP jvm_info VM version info # TYPE jvm_info gauge jvm_info{runtime="OpenJDK Runtime Environment",vendor="AdoptOpenJDK",version="11.0.8+10",} 1.0 # HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded. # TYPE jmx_config_reload_failure_total counter jmx_config_reload_failure_total 0.0 # HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds. # TYPE jvm_gc_collection_seconds summary jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 883.0 jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 133.293 jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0 jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0 # HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. # TYPE jvm_memory_pool_allocated_bytes_total counter jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 6.76448896E8 jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 1.345992784E10 jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 9.062406160384E12 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 3.38238592E8 jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 1.6919822336E10 jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 1.41419488E8 jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 1.141665096E9 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 3544448.0   I do see the values in Splunk, but especially for the last metric "jvm_memory_pool_allocated_bytes_total" the label of which pool is lost in splunk. Is this intentional or am I missing something. The getting started page for metrics also has no information on where those labels are stored and how I could query based on them (https://docs.splunk.com/Documentation/Splunk/latest/Metrics/GetStarted)   tia,     Jörg ... View more