About lee_melvin

lee_melvin · ‎12-15-2022

If this is still an issue for you a year and a half later, please take a look at my (redundant) question "Splunk_TA_ontap OntapPerf.py only fetches a subset of volumes". There's a trivial modification to the OntapPerf.py file in the add-on, that will collect data for all volumes.

lee_melvin · ‎12-15-2022

I finally ran this one to ground, after several attempts over the last few months. The short form here is to modify Splunk_TA_ontap/bin/ta_ontap/OntapPerf.py, and change line ~500 from next_tag = ET.tostring(next_tag.getchildren()[0]) to next_tag = ET.tostring(next_tag.getchildren()[0], encoding="unicode") As far as I can tell, the default encoding for the tostring function changed between XML modules in the Python 2 (Splunk 7.x) and Python 3 (Splunk 8.x) releases. It used to default to some basic string, and changed to a byte stream. If you look at the tostring definition in splunk/lib/python3.7/xml/etree/ElementTree.py line ~1134: stream = io.StringIO() if encoding == 'unicode' else io.BytesIO() I'm guessing that didn't happen in the earlier version of the xml module. Really hoping that Splunk will develop a (supported) complete replacement ONTAP add-on that leverages REST API instead of the current unsupported add-on using ONTAPI...

lee_melvin · ‎05-19-2022

The "Splunk Add-on for NetApp Data ONTAP" is only fetching performance information for the first 50 volumes on a cluster. Changing the "perf_chunk_size_cluster_mode" value in ta_ontap_collection.conf will vary the number -- if I set it to 53, I'll get performance data for the first 53 alphabetic volumes on the cluster. You can't set this arbitrarily large, if I put it to 10000, I get a failure on the data collection. The chunking mechanism is part of OntapPerf.py, and normally would iterate over multiple queries until it collected data for all volumes. This worked for years, but has been broken for several months now. May or may not align with our upgrade to Splunk Enterprise 8.2.2 and Python 3. Add-on is latest version (3.0.2). Filers are running ONTAP 9.7. I went back through the install manual and verified all the steps and add-on data. Inventory/capacity information works without issue, it is just performance metrics that are a problem. OntapPerf.py throws log warnings "No instances found for object type volume"... from line 461 in the script. It seems like the "next_tag" mechanism in the script is failing, but I can't work out how to run OntapPerf.py from command line, I don't know how to troubleshoot any further. Splunk_TA_ontap shows as "unsupported" and developed by "Splunk Works". Last release was June 2021. I could really use some pointers on how to resolve this, or how I could move forward troubleshooting it myself.

lee_melvin · ‎02-07-2017

The 'streamstats | stats last(*)' construct is the concept I was missing. I ended up with a search rather like this: source=backup OR source=volume | eval fs_path=path | eval backup_path=ndmp_path | eval path=coalesce(fs_path, backup_path) | sort -path source | streamstats current=f reset_on_change=t values(path) as allpaths by server | eval idx=mvfind(allpaths, "^" + path) | eval match_path=if(isnotnull(idx), mvindex(allpaths, idx), path) | stats last(*) as * by server match_path | table server path fs_path backup_path I still have some details to work out, but its workable. Thanks!

lee_melvin · ‎01-12-2017

I want to group events describing backup job status with other events describing the volumes being backed up. The data conceptually looks like this: source=backup server=a path=/vol/engineering status=successful level=0 size_gb=23 source=backup server=b path=/scratch test=failed level=0 source=backup server=z path=/regression status=successful level=1 size_gb=1 and source=volume server=a ndmp_path=/vol/engineering owner=bob source=volume server=b ndmp_path=/test owner=jill source=volume server=c path=/vs0/hr owner=jack source=volume server=d path=/scratch_nobackup owner=zack I want to see what is getting backed up, what isn't and whether something is getting backed up that I don't have in my volume inventory. I can do that with something like a stats last() or join server path , ie source=backup OR source=volume | status last(*) as * by server, path and then identify protected data (owner=* size_gb=*) or unprotected data (owner=* NOT size_gb=*) or volumes that are backed up but not in inventory (size_gb=* NOT owner=*) . The problem I'm running into is that the backup path is sometimes a subdirectory of the volume inventory path: source=backup server=q path=/hr/important/data status=successful level=0 size_gb=2 source=volume server=q path=/hr owner=ellen or source=backup server=m path=/flower/database/.snapshot/daily status=successful level=0 size_gb=3 source=volume server=m path=/flower/database owner=don I need to group these together -- same server, and volume path is a prefix of the backup path. But there is no stats last(*) as * by server, one_is_a_prefix_of_the_other(path) . I've looked into stats, join, and transaction and haven't turned up anything obvious. I can't extract the first field of the path and group by that, because sometimes the volume path is depth 1 (/volume) and sometimes it is depth 2 (/server_name/volume or /vol/volume). I was able to get close by generating a lookup file for the volumes, then using wildcard functionality in transforms.conf to pull volume fields into the backup data. But that has some complications of its own that I'd rather avoid. Thanks! Lee

lee_melvin · ‎09-30-2016

I came back to needing to do this again, dug around, and found a trivial solution (https://answers.splunk.com/answers/47101/complete-a-timechart-with-a-total-column.html) -- do the timechart, then just use | addtotals to get what I needed. Sigh.

lee_melvin · ‎09-30-2016

Changing the timeout from 30s to 60s dropped my read timeout error rate from ~25/hr to about 3/hr. I'm not sure if I'm solving a problem (WAN data collection just needs more time) or masking some other issue that needs solving, but it reduces my gaps in data collection. I've set a standard of CONNECTION_TIMEOUT = 120 for all existing and future DCNs in our environment.

lee_melvin · ‎09-22-2016

I'm getting read timeout errors for NetApp data collection using the Splunk App for NetApp Data ONTAP, when collecting from filers over a WAN connection. In the "Connection Problems in the Past Hour" report, I get messages like this: 2016-09-22 07:36:41,912 ERROR [ta_ontap_collection_worker://zeta:4535] [QuotaHandler] Problem collecting ontap:quota data from server=somefiler.somesite.mentorg.com : ('The read operation timed out',) Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/handlers.py", line 129, in run results = qa.run() File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapInventory.py", line 106, in run return self.run7Mode() File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapInventoryQuota.py", line 20, in run7Mode self.aggregate_results(data, 'quota-report-iter-start', secondLevelDictArray) File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapInventory.py", line 74, in aggregate_results for x in gen: File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 521, in query7ModeGen response = naElementToDict(self.queryApi(api, OntapClient.projectResponse)) File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/OntapClient.py", line 418, in queryApi response = self.connection.invoke_elem(api) File "/opt/splunk/etc/apps/Splunk_TA_ontap/bin/ta_ontap/NetApp/NaServer.py", line 483, in invoke_elem xml_response = response.read() File "/opt/splunk/lib/python2.7/httplib.py", line 593, in read s = self.fp.read() File "/opt/splunk/lib/python2.7/socket.py", line 355, in read data = self._sock.recv(rbufsize) File "/opt/splunk/lib/python2.7/ssl.py", line 734, in recv return self.read(buflen) File "/opt/splunk/lib/python2.7/ssl.py", line 621, in read v = self._sslobj.read(len or 1024) SSLError: ('The read operation timed out',) The problem happens with filers at multiple sites, but is only occasional - mostly the data is collected as expected. WAN saturation does not appear to be an issue. The lowest level of (apparent) SSL configurable timeout I could find is in Splunk_TA_ontap/bin/ta_ontap/NetApp/OntapClient.py, it has a CONNECTION_TIMEOUT=30 setting. I have changed that to 60 and will see if it alleviates the issue. We have filers at 30+ sites, but the DCNs are all in our primary datacenter. Would it be better to have local DCNs and not collect data over the WAN? If I create a bunch of site-local DCNs, will the scheduler automatically know what DCN is "closest" to a filer (or can that be manually configured)? Thanks! Lee

lee_melvin · ‎07-29-2016

We had the same issue upgrading from 6.2 to 6.4; a bunch of our filers dropped out of data collection. To put a little extra detail on it, for 7-mode at least, this means setting options httpd.admin.ssl.enable on options tls.enable on

lee_melvin · ‎03-06-2015

Based on ramdaspr's comment, I fumbled my way to this solution: sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time This solves my immediate need.

lee_melvin · ‎03-06-2015

As I understand it, per_day() is kind of like span=1d -- if I poll a volume 24 times in 1 day, and have 24 corresponding events that say volume=foo usedGB=35 , per_day(usedGB) isn't going to return '35', it will return 35*24, which isn't what I need. Conceptually what I want is "for each day, do a dedup filer,volume | stats sum(usedGB) '. After your comment I spent some time digging into timechart and per_day documentation, and then on bucket and stats. I think I have arrived at a working (but probably not optimal) solution, which looks like sourcetype=<blah> | bucket span=1d _time | stats max(usedGB) as usedGB by _time, filer, volume | stats sum(sizeGB) by _time The 'bucket' forces all the events to have discrete daily times, the stats max serves to deduplicate identical filer/volume events that occur in the same time bucket, and stats sum gets me the total I want per day. I end up with nested " stats | stats " where before I was attempting " timechart | timechart ". It looks like the timechart wanted to convert all the component fields into columns (and beyond a certain number of columns it starts lumping everything into 'OTHER'), but stats puts everything into discrete events, which ended up being a little more flexible. One of the takeaways for me is that "timechart foo" is not identical to/short for "stats foo by _time". per_day() didn't solve my issue, but started me towards a resolution - thanks!

lee_melvin · ‎03-05-2015

If I have a number of storage devices, each with a number of volumes, and every hour I am querying the used capacity of each volume, and I want to graph the total used capacity for all storage devices by day over the last two weeks, how do I construct a search string for that? _time=1am filer=bob volume=source usedGB=35 _time=1am filer=jill volume=build usedGB=10 _time=1am filer=jill volume=test usedGB=50 _time=2am filer=bob volume=source usedGB=35 _time=2am filer=jill volume=build usedGB=11 _time=2am filer=jill volume=test usedGB=49 To get the current usage I do something like sourcetype=<blah> | dedup filer, volume | stats sum(usedGB) But if I want to chart over time, I get confused. I could do something like sourcetype=<blah> | timechart sum(usedGB) but if the timechart span is larger than the filer polling period, I'm going to end up adding the volume sizes into the total multiple times for each time bucket, which isn't correct. Then I wanted to do something like sourcetype=<blah> | timechart last(usedGB) as usedGB by filer, volume | timechart sum(usedGB) but you can't timechart by two fields. Then I tried to construct a single unique field sourcetype=<blah> | eval unique=filer.":".volume | timechart last(usedGB) as usedGB by unique| timechart sum(usedGB) which seemed both clunky and promising, but didn't generate any data. I don't think the nested timecharts are doing what I want. Finally I started brainstorming around combining bucket and stats last(), but had no luck. This feels like a problem with an obvious answer that I'm just not seeing. Any help appreciated. Lee

Posts	12
Solutions	1
Karma Given	1
Karma Received	1
Member Since	‎05-28-2014

Online Status	Offline
Date Last Visited	‎12-15-2022 04:09 PM

Splunk_TA_ontap OntapPerf.py only fetches a subset...

How to group events by a common directory prefix

Splunk App for NetApp Data ONTAP: Why am I receivi...

How to graph a (non-cumulative) total of fields?

Re: Splunk TA for NetApp Data ONTAP - insufficient...

Re: Splunk_TA_ontap OntapPerf.py only fetches a su...

Splunk_TA_ontap OntapPerf.py only fetches a subset...

Re: How to group events by a common directory pref...

How to group events by a common directory prefix

Re: How to graph a (non-cumulative) total of field...

Re: Splunk App for NetApp Data ONTAP: Why am I rec...

Splunk App for NetApp Data ONTAP: Why am I receivi...

Re: Splunk App for NetApp Data ONTAP: After upgrad...

Re: How to graph a (non-cumulative) total of field...

Re: How to graph a (non-cumulative) total of field...

How to graph a (non-cumulative) total of fields?