Collecting local load balance metrics using the iControl API dies works for a little while the dies with SSLError

[rmoss84]

Whenever we turn on the the API collection for local load balancer metrics, it will work for a few minutes to a couple of hours, but ultimately it will stop collecting the data. Every time it does this, we get the below error message:

2019-04-22 11:06:22,765 ERROR pid=33429 tid=Thread-46 file=f5_bigip_scheduler.py:run:177 | Error in F5BigIPWorker.run() - Traceback:
Traceback (most recent call last):
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/f5_bigip_scheduler.py", line 167, in run
items=query.fetch()
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/F5_iControl_Template.py", line 77, in fetch
return [event for partition in partitions for event in self.fetch(partition)]
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/F5_iControl_Template.py", line 136, in _fetch
dataAgainst=F5_iControl_API.run(bigip=bigip, api=aApiAgainst, params=dataWith)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/F5_iControl_API.py", line 90, in run
return method(params) if params else method()
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/bigsuds.py", line 416, in wrapped_method
result = method(*args, **kwargs)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/client.py", line 542, in __call_
return client.invoke(args, kwargs)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/client.py", line 637, in send
reply = transport.send(request)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/transport/https.py", line 64, in send
return HttpTransport.send(self, request)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/transport/http.py", line 77, in send
fp = self.u2open(u2request)
File "/app/splunk/etc/apps/Splunk_TA_f5-bigip/bin/Splunk_TA_f5_bigip/suds/transport/http.py", line 118, in u2open
return url.open(u2request, timeout=tm)
File "/app/splunk/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/app/splunk/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/app/splunk/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/app/splunk/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/app/splunk/lib/python2.7/urllib2.py", line 1201, in do_open
r = h.getresponse(buffering=True)
File "/app/splunk/lib/python2.7/httplib.py", line 1121, in getresponse
response.begin()
File "/app/splunk/lib/python2.7/httplib.py", line 438, in begin
version, status, reason = self._read_status()
File "/app/splunk/lib/python2.7/httplib.py", line 394, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/app/splunk/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
File "/app/splunk/lib/python2.7/ssl.py", line 766, in recv
return self.read(buflen)
File "/app/splunk/lib/python2.7/ssl.py", line 653, in read
v = self._sslobj.read(len)
SSLError: ('The read operation timed out',)

We have a case opened with F5, but they are pretty much useless. Wondering if anyone else has had this issue and what they did to resolve this?

[codebuilder]

You need to increase the timeout settings on your F5. I'm not sure where these are configured on an F5, but in haproxy I have the following values configured:

timeout connect         5000
timeout client            50000
timeout server          50000