Volume detail not showing data in Monitoring Console

[mykol_j]

v9.2.0.1

Monitoring Console in Splunk manager is not displaying volume information. All panels say "Search is waiting for input...". When I open the search of a given panel, the query it opens to is an "All time" query with "undefined" in the query box...

I'm trying to monitor volume (and index) size/space using the Management console, Indexing, Volume Details. ../app/splunk_monitoring_console/volume_detail_deployment...

The dashboards/panels populate fine when looking at index data, but are empty when trying to view volume data. Index panels have a nice rest query, and the Volume panels are all "undefined".

Fixing thoughts?

Cheers,

[glc_slash_it]

Hi,

do you have a distributed architecture or just single instance?

Did you set the volume settings on indexes.conf?

[volume:primary]
path = /path/to/storage/partition
maxVolumeDataSizeMB = 5000000

[mykol_j]

I'm not following you...

It's a cluster of indexers (4) and a single Management node.

...and I'm failing to see how setting the volume size affects the missing queries in the management dashboard. But to answer the questions: of course I did. I have multiple volumes per best practices.

[glc_slash_it]

The missing information can be the result of one or several missing/wrong configurations either on MC or IDXs, that will depend on the architure you have. So it's important to frame your case.

That dashboard displays "Search is waiting for input" beacase there are probably missing tokens like the Volume dropdown, right? As an example, my system don't have any volumes defined and so the "Volume" dropdown will not populate, preventing the dashboard from running searches, and thus showing Undefined.

Did you setup the Monitoring Console in that Managment Node? Can it "see" all the Indexers in Settings > Distributed Search?

Is the info about individual indexes accurate?

If you go to MC > Settings > General Setup, does all instances show with correct information?

https://docs.splunk.com/Documentation/Splunk/9.2.1/DMC/Configureindistributedmode

[mykol_j]

ah, gotcha. Yes, it's configured, setup is correct, server roles are set, and I use it often for various things -- I can see data in pretty much every other dashboard, and even in the "Index Detail: Deployment" -- it *does* show some volume information, as does "Indexes and Volumes: Deployment". In "opening in search" pretty much any panel in the monitoring console, I can see the query and the macro or rest it uses. But in these volume detail pages, they all are "undefined".

The info reporting on the individual indexes is correct -- I use it to trim and set limits on various indexes.

I do see in the "Volume Detail: Instance" -- I each indexer populated in the dropdown, but the Volume (token) is empty.

To recap:  all my dashboards in Monitoring Console on my Management server have data except for:  VolumeDetail: Instance and VolumeDetail: Deployment.

Honestly, if it's NOT configured correctly (the management server/console), then I'm not sure what to fix. I know this doesn't always mean a Good Thing, but I have been using Splunk since v3.x.

[mykol_j]

Holy Cow!

Per that document, I tried enabling:

mc_auto_config = enabled

...and it removed all my indexers from the cluster. Good times.

I think I'll just learn to live without those volume dashboards, wouldn't be the first time I had to ignore missing functionality.

Pro tip:  don't bother taking the advanced troubleshooting class at Splunk.conf -- didn't prepare me for anything useful...