Why is my search using "mcollect" command causing the following error: "Error in 'mcollect' command: Must specify a valid metric index"?

[ben_leung]

In my query before, I was using the outputcsv search command, and then I had a monitoring input stanza to upload it to my metrics index.

I then took out the outputcsv command and started using mcollect.

Not sure why, but the metrics index is not valid when it recieved metrics from a different method.

....  | mcollect index=metrics-index

And on my indexes.conf, the settings is configured to be metrics:

[metrics-index]
datatype = metric

[anem]

index=xyz source=abc |table coloumnone coloumtwo |mcollect index=metric_index split=true coloumtwo

above is the example hope it helps

[ben_leung · Accepted solution]

Okay so to resolve the issue, setup the metrics index on the search head cluster.
Missing the indexes.conf on our search head cluster. Overlooked this because we have indexes defined on our indexers.
Also because the previous method of using outputcsv and a monitoring stanza did not require any index definition on search heads.

[splunkIT]

In a distributed splunk environment, the search head also needs to have stub index of the same name and datatype for any of the collect (ie. collect, mcollect, meventcollect). Note: for metric index, you will need to specify datatype = metric in the relevant index.conf stanza.

[ben_leung]

Right, just couldn't find a doc for version 7.1.2 about that.

[DalJeanis]

So, here's some triage steps.

---

1) Put quotes around your metrics index name and submit again. If that fixes the problem, then the hyphen is being incorrectly interpreted as a minus sign. Skip the remainder of the steps.

---

2A) Create a new, empty metrics index "junkmetrics" with no special characters in the name.

2B) Try your query with mcollect with that new index.

If that works, then the problem is either the name with the hyphens, or the prior data in the index.

2C) Create a junk index with hyphens in the name and try again. (It's supposed to be allowed as an index name, but none of the examples have hyphens.)

If that works, then the prior data is the likely culprit.

---

3) If that (2A) does not work, then those two items are off the table, and you know it is probably something with your search language itself. You've already tested the index name, so just report back and let us know, and we'll give you the next tranche of guesses.

[ben_leung]

I feel that the output of the results may not be right since the method is different.

| table _time, value, a_dimension, metric_name | mcollect index=test_metric

_time value a_dimension metric_name
2018-09-12 12:16:01.100 1.45 view_point view.point.metrics
2018-09-12 12:19:01.100 0.9 view_point view.point.metrics
2018-09-12 12:21:01.100 3.41 view_point view.point.metrics

Field value is a double. If I try to rename value to _value, still does not work.

[ben_leung]

This is a distributed search environment with search head clustering.