Splunk_TA_ontap OntapPerf.py only fetches a subset of volumes

[lee_melvin]

The "Splunk Add-on for NetApp Data ONTAP" is only fetching performance information for the first 50 volumes on a cluster.  Changing the "perf_chunk_size_cluster_mode" value in ta_ontap_collection.conf will vary the number -- if I set it to 53, I'll get performance data for the first 53 alphabetic volumes on the cluster.   You can't set this arbitrarily large, if I put it to 10000, I get a failure on the data collection.  

The chunking mechanism is part of OntapPerf.py, and normally would iterate over multiple queries until it collected data for all volumes.  This worked for years, but has been broken for several months now.  May or may not align with our upgrade to Splunk Enterprise 8.2.2 and Python 3.

Add-on is latest version (3.0.2).  Filers are running ONTAP 9.7.  I went back through the install manual and verified all the steps and add-on data.  Inventory/capacity information works without issue, it is just performance metrics that are a problem.

OntapPerf.py throws log warnings "No instances found for object type volume"... from line 461 in the script.  

It seems like the "next_tag" mechanism in the script is failing, but I can't work out how to run OntapPerf.py from command line, I don't know how to troubleshoot any further.

Splunk_TA_ontap shows as "unsupported" and developed by "Splunk Works".   Last release was June 2021.  I could really use some pointers on how to resolve this, or how I could move forward troubleshooting it myself.   

[lee_melvin · Accepted solution]

I finally ran this one to ground, after several attempts over the last few months.

The short form here is to modify Splunk_TA_ontap/bin/ta_ontap/OntapPerf.py, and change line ~500 from

next_tag = ET.tostring(next_tag.getchildren()[0])

to

next_tag = ET.tostring(next_tag.getchildren()[0], encoding="unicode")

As far as I can tell, the default encoding for the tostring function changed between XML modules in the Python 2 (Splunk 7.x) and Python 3 (Splunk 8.x) releases.  It used to default to some basic string, and changed to a byte stream.  If you look at the tostring definition in splunk/lib/python3.7/xml/etree/ElementTree.py line ~1134:

   stream = io.StringIO() if encoding == 'unicode' else io.BytesIO()

I'm guessing that didn't happen in the earlier version of the xml module.

Really hoping that Splunk will develop a (supported) complete replacement ONTAP add-on that leverages REST API instead of the current unsupported add-on using ONTAPI...