Hi All,
Great little app Speedtest (https://splunkbase.splunk.com/app/3530/) which has been working perfectly for the last few years. Until my ISP upgraded the network last week and now my upload metrics are off. Only when Splunk runs the script. If I manually run it it works perfectly.
It's super odd.
Whenever Splunk runs it as a scheduled scripted input the upload value returns ~3.5Mbps.
When I manually run the script from the same server it returns ~21Mbps
{"client": {"rating": "0", "loggedin": "0", "isprating": "3.7", "ispdlavg": "0", "ip": "193.116.81.55", "isp": "TPG Internet", "lon": "153.0215", "ispulavg": "0", "country": "AU", "lat": "-27.4732"}, "bytes_sent": 27787264, "download": 251570694.55484477, "timestamp": "2021-02-07T23:46:05.028139Z", "share": null, "bytes_received": 315446696, "ping": 16.562, "upload": 21321234.56928962, "server": {"latency": 16.562, "name": "Brisbane", "url": "http://brs1.speedtest.telstra.net:8080/speedtest/upload.php", "country": "Australia", "lon": "153.0278", "cc": "AU", "host": "brs1.speedtest.telstra.net:8080", "sponsor": "Telstra", "lat": "-27.4728", "id": "2604", "d": 0.62311775977947}}
Running it from desktop it returns ~21Mbps - https://www.speedtest.net/result/10890590348
I've tried playing around with the schedule but it doesn't seem to help.
Any ideas?
This was working just fine until my ISP upgraded the network last week. Previously I have 50Mbps down and 23Mbps up and the scripted input was accurately reflecting the measurements.
Pinging author @markhill1 in case he has any ideas.
The script for the input speedtest-cli.py actually stopped working completely around May 2021. I managed to download the latest version from (see wget command)
https://github.com/sivel/speedtest-cli
Then rename from speedtest-cli to speedtest-cli.py
It's BACK!!!!!!
and it doesn't have the 1/6 speed issues I used to have.
@markhill1 maybe if you can release an updated version with the latest version of this script from Github it might help others. If you dare run the brutal gauntlet of the Splunk's app inspector.
The script for the input speedtest-cli.py actually stopped working completely around May 2021. I managed to download the latest version from (see wget command)
https://github.com/sivel/speedtest-cli
Then rename from speedtest-cli to speedtest-cli.py
It's BACK!!!!!!
and it doesn't have the 1/6 speed issues I used to have.
@markhill1 maybe if you can release an updated version with the latest version of this script from Github it might help others. If you dare run the brutal gauntlet of the Splunk's app inspector.
Hi Cameron, How odd!
I'll take a look and see if I can find out what's happening.
Weird.
The plot thickens.
Maybe something to do with Splunk's Python?????
I'm running Splunk 8.1.2 (build 545206cc9f70)
Edit: I think I found the issue
Manually run with
Some debug code I added with results for both Python 2 and 3 run
###########################################################
# Python Code changes - Added debugs where it says TESTING
###########################################################
def upload(self, callback=do_nothing, pre_allocate=True):
"""Test upload speed against speedtest.net"""
sizes = []
for size in self.config['sizes']['upload']:
for _ in range(0, self.config['counts']['upload']):
sizes.append(size)
# request_count = len(sizes)
request_count = self.config['upload_max']
requests = []
for i, size in enumerate(sizes):
# We set ``0`` for ``start`` and handle setting the actual
# ``start`` in ``HTTPUploader`` to get better measurements
data = HTTPUploaderData(
size,
0,
self.config['length']['upload'],
shutdown_event=self._shutdown_event
)
if pre_allocate:
data.pre_allocate()
requests.append(
(
build_request(self.best['url'], data, secure=self._secure),
size
)
)
def producer(q, requests, request_count):
for i, request in enumerate(requests[:request_count]):
thread = HTTPUploader(
i,
request[0],
start,
request[1],
self.config['length']['upload'],
opener=self._opener,
shutdown_event=self._shutdown_event
)
thread.start()
q.put(thread, True)
callback(i, request_count, start=True)
finished = []
def consumer(q, request_count):
while len(finished) < request_count:
thread = q.get(True)
while thread.isAlive():
thread.join(timeout=0.1)
finished.append(thread.result)
callback(thread.i, request_count, end=True)
q = Queue(self.config['threads']['upload'])
prod_thread = threading.Thread(target=producer,
args=(q, requests, request_count))
cons_thread = threading.Thread(target=consumer,
args=(q, request_count))
start = timeit.default_timer()
prod_thread.start()
# TESTING
printer('Upload Start:%d\n' % start, debug=True)
cons_thread.start()
while prod_thread.isAlive():
prod_thread.join(timeout=0.1)
while cons_thread.isAlive():
cons_thread.join(timeout=0.1)
stop = timeit.default_timer()
# TESTING
printer('Upload Stop:%d\n' % stop, debug=True)
self.results.bytes_sent = sum(finished)
# TESTING
printer('Bytes sent:%s\n' % finished, debug=True)
printer('Bytes sent:%d\n' % self.results.bytes_sent, debug=True)
self.results.upload = (
(self.results.bytes_sent / (stop - start)) * 8.0
)
# TESTING
printer('Upload Value:%d\n' % self.results.upload, debug=True)
return self.results.upload
###########################################################
# Python 2 Run
###########################################################
DEBUG: Upload Start:1613343269
.................................................................DEBUG: Upload Timeout: result=114688
DEBUG: Upload Timeout: result=458752
DEBUG: Upload Timeout: result=974848
DEBUG: Upload Timeout: result=483328
...............................
DEBUG: Upload Stop:1613343279
DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 974848, 483328, 458752, 114688, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DEBUG: Bytes sent:26148864
DEBUG: Upload Value:20746227
Upload: 20.75 Mbit/s
DEBUG: Results:
{'client': {'rating': '0', 'loggedin': '0', 'isprating': '3.7', 'ispdlavg': '0', 'ip': '193.116.x.x', 'isp': 'TPG Internet', 'lon': '153.0215', 'ispulavg': '0', 'country': 'AU', 'lat': '-27.4732'}, 'bytes_sent': 26148864, 'download': 0, 'timestamp': '2021-02-14T22:54:27.902614Z', 'share': None, 'bytes_received': 0, 'ping': 14.774, 'upload': 20746227.40552687, 'server': {'latency': 14.774, 'name': 'Brisbane', 'url': 'http://qld.speedtest.exetel.com.au:8080/speedtest/upload.php', 'country': 'Australia', 'lon': '153.0251', 'cc': 'AU', 'host': 'qld.speedtest.exetel.com.au:8080', 'sponsor': 'Exetel', 'lat': '-27.4698', 'id': '13276', 'd': 0.5187222967744429}}
###########################################################
# Python 3 Run
###########################################################
.DEBUG: Upload Start:224944
..DEBUG: Upload Timeout: result=524288
..DEBUG: Upload Timeout: result=524288
..DEBUG: Upload Timeout: result=524288
..DEBUG: Upload Timeout: result=524288
.............................................................................................
DEBUG: Upload Stop:224955
DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
DEBUG: Bytes sent:2097152
DEBUG: Upload Value:1574428
Upload: 1.57 Mbit/s
DEBUG: Results:
{'download': 0, 'upload': 1574428.9733984023, 'ping': 13.135, 'server': {'url': 'http://brs1.speedtest.telstra.net:8080/speedtest/upload.php', 'lat': '-27.4728', 'lon': '153.0278', 'name': 'Brisbane', 'country': 'Australia', 'cc': 'AU', 'sponsor': 'Telstra', 'id': '2604', 'host': 'brs1.speedtest.telstra.net:8080', 'd': 0.62311775977947, 'latency': 13.135}, 'timestamp': '2021-02-14T22:55:17.200248Z', 'bytes_sent': 2097152, 'bytes_received': 0, 'share': None, 'client': {'ip': '193.116.x.x', 'lat': '-27.4732', 'lon': '153.0215', 'isp': 'TPG Internet', 'isprating': '3.7', 'rating': '0', 'ispdlavg': '0', 'ispulavg': '0', 'loggedin': '0', 'country': 'AU'}}
[root@primary bin]#
I've temporarily resolved by forcing python2 in inputs.conf
[script://$SPLUNK_HOME/etc/apps/speedtest/bin/speedtest-cli.py --json]
disabled = 0
interval = 5,35 * * * *
python.version = python2
Thanks Mark. Happy to run some more tests if you need me to.
It's pretty odd though I have no idea how you would even begin to troubleshoot it or how it would behave like this only when run by Splunk.