I am running 6.3.1 on my search head and 6.3.1 on my 3 indexers. I can see the on the resource usage per instance the historic values of the my search heads "Median Physical Memory Usage by Process Class" but when i ask for any of the indexers no data is available.
On the indexers i can see the file /opt/splunk/var/log/introspection/resource_usage.log but even though in the data files the directory are selected for consumption only the kvstore.log is present in the _introspection index.
In the _internal index I can see an entry when I disable and then re-enable these files as follows
10-24-2016 14:29:29.024 +0100 ERROR TailReader - Ignoring path="/opt/splunk/var/log/introspection/resource_usage.log" due to: Invalid indexed extractions configuration - see prior error messages
I was seeing the same behavior. Both resource_usage.log and disk_objects.log were not getting collected, so many DMC panels were blank (sometimes DMC can be tricky). I found that somehow we had two invalid props.conf entries in our _cluster app. To find this:
In the splunkd.log, look for the log entry that proceedes the TailReader error you are seeing above, in our case we saw:
`01-20-2017 13:04:38.063 -0500 ERROR IndexedExtractionsConfig - Invalid value='' for parameter='INDEXED_EXTRACTIONS'.
01-20-2017 13:04:38.063 -0500 ERROR TailReader - Ignoring path="C:\Program Files\Splunk\var\log\introspection\resource_usage.log.3" due to: Invalid indexed extractions configuration - see prior error messages
Notice the invalid parameter is INDEXED_EXTRACTIONS and it claims we have a value of ''.
2. On one of your peer indexers, run the following command. You may want to redirect the output to a file to make it easier to view. splunk btool props list --debug
3. In the output from above, find the line where the invalid parameter is set. In your case as in ours it should be under heading [splunk_resource_usage] and [splunk_disk_objects]
4. Once you find the line, it should point you to the config file that has the bad value. In our case this was in etc\slave-apps\_cluster\local\props.conf, so we just fixed the copy of this file in the master-apps on the cluster master and redistributed the cluster bundle. For some reason we had the following 4 lines in this file: [splunk_disk_objects]
We removed all four lines.
5. The change did not cause the indexers to restart, so I had to manually run a rolling restartsplunk rolling-restart cluster-peers`. Once the peers restarted, the all the panels that rely on these logs started to render data.