using mcollect to collect metric data

[_pravin]

Hi,

I am using mcollect to collect data from certain metrics into another metric index. I have created the new metric index in the search head and also in the indexer clusters.

The command looks something like this, but whenever I run the command, I get an error 'No results to summary index'. 

| mpreview index=metrics_old target_per_timeseries=5 filter="metric_name IN ( process.java.gc.collections) env IN (server_name:port)" 
| mcollect index=metrics_new

 Is there something I'm doing wrong when using the mcollect command?

Please advise. Thanks in advance.

Regards,

Pravin

[livehybrid]

Hi @_pravin 

The error "No results to summary index" suggests that the first part of your query didnt return any events, or didnt return events which could be interpeted as a metric.

Can you run the first line to confirm you are getting events returned?

Can you also please confirm that you have permission to read/write to metrics_new and that (as the name suggests ) it is definitely a metric index?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[_pravin]

Hi @livehybrid ,

To answer your questions in order,

1. I can get events, and it works when I run the first line.
2. I am logged in as an admin, and I created the index, so I have permission to read and write to the index.

Data is available, but not written to the new index.

info : The limit has been reached for log messages in info.csv. 65149 messages have not been written to info.csv. Refer to search.log for these messages or limits.conf to configure this limit.
warn : No results to summary index.

Once I complete the execution, I get the above error message in the job inspector.

Thanks,

Pravin

[LAME-Creations]

It looks like your mcollect command isn’t writing data to the metrics_new index, despite mpreview returning results. The "No results to summary index" error and the info.csv limit warning suggest a couple of potential issues. Here’s how you can troubleshoot and fix it:

1. Verify mcollect Syntax:
   • Ensure the fields required by mcollect are present. For metrics data, mcollect expects _value (numeric metric value), metric_name, and any dimensions (e.g., env, server_name). Your mpreview query should already include these, but confirm the output includes _value and metric_name:process.java.gc.collections.
   • Try adding | fields metric_name, _value, env before mcollect to explicitly pass only the required fields:
     spl

     | mpreview index=metrics_old target_per_timeseries=5 filter="metric_name IN (process.java.gc.collections) env IN (server_name:port)"| fields metric_name, _value, env
     | mcollect index=metrics_new

2. Check Index Configuration:
   • Confirm metrics_new is a metrics index (not an event index) on both the search head and indexers. Run | eventcount summarize=false index=metrics_new to verify the index exists and is accessible.
   • Ensure the index is not full or disabled. Check $SPLUNK_HOME/etc/apps/<app>/local/indexes.conf on the indexers for metrics_new settings and verify frozenTimePeriodInSecs or maxTotalDataSizeMB aren’t causing issues.
3. Address info.csv Limit Warning:
   • The warning about info.csv reaching its limit (65,149 messages) suggests the search is generating excessive log messages, which might indicate a problem with the mpreview output or internal processing. This can sometimes prevent mcollect from writing results.
   • Increase the info.csv limit in $SPLUNK_HOME/etc/system/local/limits.conf:
     ini

     [search]
     max_messages_per_info_csv = 100000

   • Alternatively, reduce the volume of data processed by narrowing the time range or tightening the filter in mpreview (e.g., specify a single env value).
4. Inspect Search Logs:
   • The job inspector mentions errors in search.log. Check $SPLUNK_HOME/var/log/splunk/search.log for details on why mcollect is failing. Look for errors related to index access, field mismatches, or data format issues.
   • Common issues include missing _value fields or incompatible data types for mcollect.
5. Test with a Simpler Query:
   • To isolate the issue, try a minimal mcollect command:
     spl

     | mpreview index=metrics_old target_per_timeseries=1 filter="metric_name=process.java.gc.collections"| mcollect index=metrics_new

   • If this works, gradually add back the env filter to identify where the issue arises.

Next Steps:

• Run the modified query with | fields and check if data writes to metrics_new using | mpreview index=metrics_new.
• Share any specific errors from search.log or confirm if the fields (metric_name, _value, env) are present in the mpreview output for further assistance.

[_pravin · Accepted solution]

Hi @LAME-Creations ,

I figured out the problem related to writing to the indexers.  The issue was that the Search Head wasn't forwarding its data to the indexers and hence wasn't working in my case. As I created an outputs.conf on the SH, the error appeared, but the data was being written.

Thanks,

Pravin