Splunk Search

Why is my search using "mcollect" command causing the following error: "Error in 'mcollect' command: Must specify a valid metric index"?

ben_leung
Builder

In my query before, I was using the outputcsv search command, and then I had a monitoring input stanza to upload it to my metrics index.

I then took out the outputcsv command and started using mcollect.

Not sure why, but the metrics index is not valid when it recieved metrics from a different method.

....  | mcollect index=metrics-index

And on my indexes.conf, the settings is configured to be metrics:

[metrics-index]
datatype = metric

alt text

0 Karma
1 Solution

ben_leung
Builder

Okay so to resolve the issue, setup the metrics index on the search head cluster.
Missing the indexes.conf on our search head cluster. Overlooked this because we have indexes defined on our indexers.
Also because the previous method of using outputcsv and a monitoring stanza did not require any index definition on search heads.

View solution in original post

anem
Explorer

index=xyz source=abc |table coloumnone coloumtwo |mcollect index=metric_index split=true coloumtwo

above is the example hope it helps

0 Karma

ben_leung
Builder

Okay so to resolve the issue, setup the metrics index on the search head cluster.
Missing the indexes.conf on our search head cluster. Overlooked this because we have indexes defined on our indexers.
Also because the previous method of using outputcsv and a monitoring stanza did not require any index definition on search heads.

splunkIT
Splunk Employee
Splunk Employee

In a distributed splunk environment, the search head also needs to have stub index of the same name and datatype for any of the collect (ie. collect, mcollect, meventcollect). Note: for metric index, you will need to specify datatype = metric in the relevant index.conf stanza.

ben_leung
Builder

Right, just couldn't find a doc for version 7.1.2 about that.

0 Karma

DalJeanis
Legend

So, here's some triage steps.


1) Put quotes around your metrics index name and submit again. If that fixes the problem, then the hyphen is being incorrectly interpreted as a minus sign. Skip the remainder of the steps.


2A) Create a new, empty metrics index "junkmetrics" with no special characters in the name.

2B) Try your query with mcollect with that new index.

If that works, then the problem is either the name with the hyphens, or the prior data in the index.

2C) Create a junk index with hyphens in the name and try again. (It's supposed to be allowed as an index name, but none of the examples have hyphens.)

If that works, then the prior data is the likely culprit.


3) If that (2A) does not work, then those two items are off the table, and you know it is probably something with your search language itself. You've already tested the index name, so just report back and let us know, and we'll give you the next tranche of guesses.

0 Karma

ben_leung
Builder

I feel that the output of the results may not be right since the method is different.

| table _time, value, a_dimension, metric_name | mcollect index=test_metric

_time value a_dimension metric_name
2018-09-12 12:16:01.100 1.45 view_point view.point.metrics
2018-09-12 12:19:01.100 0.9 view_point view.point.metrics
2018-09-12 12:21:01.100 3.41 view_point view.point.metrics

Field value is a double. If I try to rename value to _value, still does not work.

0 Karma

ben_leung
Builder

This is a distributed search environment with search head clustering.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...