Splunk Metric Catalog REST API not returning data

[nkarthick2511]

Hi,

Im a beginner in Splunk. For an integration, Im trying to access the metric data available in Splunk using REST API.

when I try the below mentioned URI im not getting any data returned (Just getting a link to the acl)

https://<host>:<mPort>/services/catalog/metricstore/metrics

Im authorizing as a user with list_metrics_catalog capability. Please find the acl permissions for the specified REST URL in my instance below,

app- system
can_change_perms-1
can_list-1
can_share_app-1
can_share_global-1
can_share_user-0
can_write-1
modifiable-1
owner - nobody
perms
read - *
write  - admin

Using mstats , im able to view the metric data in SPlunk UI but I want to access it through REST API.

Please help.

[mattymo]

Hi nkarthick2511!

What index is the data in?

Just be sure that the user making the rest call has access to the indexes where the data resides, or use a filter that looks for the proper indexes in the api call

https://docs.splunk.com/Documentation/Splunk/7.2.1/RESTREF/RESTmetrics#Default_indexes_for_Metrics_C...
https://docs.splunk.com/Documentation/Splunk/7.2.1/RESTREF/RESTprolog#Request_and_response_details

If no metric indexes are defined with
the filter parameter, Metrics Catalog
endpoints use the default indexes
specified for the role of the user. To
review or update the default indexes
for specific roles, select Settings >
Access controls > Roles, select a
role, and review or update the Indexes
searched by default field.

FWIW, Here's how I have been playing with the API lately.

First, I am working from my Splunk cli with curl and jq installed on Linux and am calling splunkd on localhostwith my admin account. If you curl with your admin account, do you see the metrics listed?

I'll curl with silent -s and insecure -k with my splunk credentials -u admin:password and output as json ?"output_mode=json", then stuff it through jq to just get the metric_name from the json payload. See API docs for more

curl -sk -u admin:helloworld https://localhost:8089/services/catalog/metricstore/metrics?"output_mode=json"; | jq .entry[].name
"kube.cluster.cpu.limit"
"kube.cluster.cpu.request"
"kube.cluster.cpu.usage_rate"
"kube.cluster.memory.limit"
"kube.cluster.memory.request"
"kube.cluster.memory.usage"
"kube.container.cpu.limit"
"kube.container.cpu.request"
"kube.container.cpu.usage"
"kube.container.cpu.usage_rate"
"kube.container.disk.io_read_bytes"
"kube.container.disk.io_read_bytes_rate"
"kube.container.disk.io_write_bytes"
"kube.container.disk.io_write_bytes_rate"
"kube.container.filesystem.available"
"kube.container.filesystem.limit"
"kube.container.filesystem.usage"
"kube.container.memory.cache"
"kube.container.memory.limit"
"kube.container.memory.major_page_faults"
"kube.container.memory.major_page_faults_rate"
"kube.container.memory.page_faults"
"kube.container.memory.page_faults_rate"
"kube.container.memory.request"
"kube.container.memory.rss"
"kube.container.memory.usage"
"kube.container.memory.working_set"
"kube.container.restart_count"
"kube.container.uptime"
"kube.namespace.cpu.limit"

I find this preferable to messing with xml but check the API Reference because not all endpoints support all encoding, as this command will require you to have to parse the returned xml for the entity.title tags, although I am sure there are ways out there to handle that, like I did they json with jq:

...truncated for brevity
    <title>kube.cluster.cpu.request</title>
    <id>https://localhost:8089/services/catalog/metricstore/metrics/kube.cluster.cpu.request</id>
    <updated>1970-01-01T00:00:00+00:00</updated>
    <link href="/services/catalog/metricstore/metrics/kube.cluster.cpu.request" rel="alternate"/>
    <author>
      <name>system</name>
    </author>
    <link href="/services/catalog/metricstore/metrics/kube.cluster.cpu.request" rel="list"/>
    <content type="text/xml">
      <s:dict>
        <s:key name="eai:acl">
          <s:dict>
            <s:key name="app"></s:key>
            <s:key name="can_list">1</s:key>
            <s:key name="can_write">1</s:key>
            <s:key name="modifiable">0</s:key>
            <s:key name="owner">system</s:key>
            <s:key name="perms">
              <s:dict>
                <s:key name="read">
                  <s:list>
                    <s:item>admin</s:item>
                    <s:item>power</s:item>
                    <s:item>splunk-system-role</s:item>
                    <s:item>user</s:item>
                  </s:list>
                </s:key>
                <s:key name="write">
                  <s:list/>
                </s:key>
              </s:dict>
            </s:key>
            <s:key name="removable">0</s:key>
            <s:key name="sharing">system</s:key>
          </s:dict>
        </s:key>
      </s:dict>
    </content>
...truncated for brevity

Hopefully this is useful. Let me know how if you figure it out!

[nkarthick2511]

Thanks for the response.
Im trying to develop a java script script to access the splunk metrics and load it into ServiceNow for analysis. For that as a first step im using a HTTP tool like POSTMAN and trying to get the results.

Is there anything I need to change in the URL if im not using CURL.

I want to see the results decide the parameters need to be passed and then jump on to script development.