All Apps and Add-ons

Speedtest App Upload Value 1/6th of actual number

cameronjust
Path Finder

Hi All,

Great little app Speedtest (https://splunkbase.splunk.com/app/3530/) which has been working perfectly for the last few years. Until my ISP upgraded the network last week and now my upload metrics are off. Only when Splunk runs the script. If I manually run it it works perfectly.

It's super odd.

Whenever Splunk runs it as a scheduled scripted input the upload value returns ~3.5Mbps.

2021-02-08_11-21-45.jpg

When I manually run the script from the same server it returns ~21Mbps

 

{"client": {"rating": "0", "loggedin": "0", "isprating": "3.7", "ispdlavg": "0", "ip": "193.116.81.55", "isp": "TPG Internet", "lon": "153.0215", "ispulavg": "0", "country": "AU", "lat": "-27.4732"}, "bytes_sent": 27787264, "download": 251570694.55484477, "timestamp": "2021-02-07T23:46:05.028139Z", "share": null, "bytes_received": 315446696, "ping": 16.562, "upload": 21321234.56928962, "server": {"latency": 16.562, "name": "Brisbane", "url": "http://brs1.speedtest.telstra.net:8080/speedtest/upload.php", "country": "Australia", "lon": "153.0278", "cc": "AU", "host": "brs1.speedtest.telstra.net:8080", "sponsor": "Telstra", "lat": "-27.4728", "id": "2604", "d": 0.62311775977947}}

 

 

2021-02-08_11-20-04.jpg

 

Running it from desktop it returns ~21Mbps  - https://www.speedtest.net/result/10890590348

2021-02-08_11-39-15.jpg

I've tried playing around with the schedule but it doesn't seem to help.

Any ideas?

This was working just fine until my ISP upgraded the network last week. Previously I have 50Mbps down and 23Mbps up and the scripted input was accurately reflecting the measurements.

2021-02-08_11-46-55.jpg

 

Pinging author @markhill1  in case he has any ideas.

Labels (1)
Tags (2)
1 Solution

cameronjust
Path Finder

The script for the input speedtest-cli.py actually stopped working completely around May 2021. I managed to download the latest version from  (see wget command)

https://github.com/sivel/speedtest-cli

Then rename from speedtest-cli to speedtest-cli.py

 

It's BACK!!!!!!

 

and it doesn't have the 1/6 speed issues I used to have.

@markhill1 maybe if you can release an updated version with the latest version of this script from Github it might help others. If you dare run the brutal gauntlet of the Splunk's app inspector.

View solution in original post

0 Karma

cameronjust
Path Finder

The script for the input speedtest-cli.py actually stopped working completely around May 2021. I managed to download the latest version from  (see wget command)

https://github.com/sivel/speedtest-cli

Then rename from speedtest-cli to speedtest-cli.py

 

It's BACK!!!!!!

 

and it doesn't have the 1/6 speed issues I used to have.

@markhill1 maybe if you can release an updated version with the latest version of this script from Github it might help others. If you dare run the brutal gauntlet of the Splunk's app inspector.

0 Karma

markhill1
Path Finder

Hi Cameron, How odd!
I'll take a look and see if I can find out what's happening.
Weird.

Tags (1)
0 Karma

cameronjust
Path Finder

speedtest-2021-02-12_15-27-54.jpgThe plot thickens.

  1. Normal run of script - OK
  2. Run script as Splunk user - OK
  3. Run script via Splunk binary - Lower result. JSON format slightly different order too.

Maybe something to do with Splunk's Python?????

I'm running Splunk 8.1.2 (build 545206cc9f70)

 

Edit: I think I found the issue

Manually run with

  1. Python2 - OK
  2. Python3 - Lower result

speedtest-2021-02-12_15-46-12.jpg

0 Karma

cameronjust
Path Finder

Some debug code I added with results for both Python 2 and 3 run

 

###########################################################
# Python Code changes - Added debugs where it says TESTING
###########################################################


    def upload(self, callback=do_nothing, pre_allocate=True):
        """Test upload speed against speedtest.net"""

        sizes = []

        for size in self.config['sizes']['upload']:
            for _ in range(0, self.config['counts']['upload']):
                sizes.append(size)

        # request_count = len(sizes)
        request_count = self.config['upload_max']

        requests = []
        for i, size in enumerate(sizes):
            # We set ``0`` for ``start`` and handle setting the actual
            # ``start`` in ``HTTPUploader`` to get better measurements
            data = HTTPUploaderData(
                size,
                0,
                self.config['length']['upload'],
                shutdown_event=self._shutdown_event
            )
            if pre_allocate:
                data.pre_allocate()
            requests.append(
                (
                    build_request(self.best['url'], data, secure=self._secure),
                    size
                )
            )

        def producer(q, requests, request_count):
            for i, request in enumerate(requests[:request_count]):
                thread = HTTPUploader(
                    i,
                    request[0],
                    start,
                    request[1],
                    self.config['length']['upload'],
                    opener=self._opener,
                    shutdown_event=self._shutdown_event
                )
                thread.start()
                q.put(thread, True)
                callback(i, request_count, start=True)

        finished = []

        def consumer(q, request_count):
            while len(finished) < request_count:
                thread = q.get(True)
                while thread.isAlive():
                    thread.join(timeout=0.1)
                finished.append(thread.result)
                callback(thread.i, request_count, end=True)

        q = Queue(self.config['threads']['upload'])
        prod_thread = threading.Thread(target=producer,
                                       args=(q, requests, request_count))
        cons_thread = threading.Thread(target=consumer,
                                       args=(q, request_count))
        start = timeit.default_timer()
		
        prod_thread.start()

		# TESTING
        printer('Upload Start:%d\n' % start, debug=True)

        cons_thread.start()
        while prod_thread.isAlive():
            prod_thread.join(timeout=0.1)
        while cons_thread.isAlive():
            cons_thread.join(timeout=0.1)


        stop = timeit.default_timer()

		# TESTING
        printer('Upload Stop:%d\n' % stop, debug=True)

        self.results.bytes_sent = sum(finished)

		# TESTING
        printer('Bytes sent:%s\n' % finished, debug=True)
        printer('Bytes sent:%d\n' % self.results.bytes_sent, debug=True)

        self.results.upload = (
            (self.results.bytes_sent / (stop - start)) * 8.0
        )

		# TESTING
        printer('Upload Value:%d\n' % self.results.upload, debug=True)

        return self.results.upload







###########################################################
# Python 2 Run
###########################################################

DEBUG: Upload Start:1613343269

.................................................................DEBUG: Upload Timeout: result=114688

DEBUG: Upload Timeout: result=458752

DEBUG: Upload Timeout: result=974848

DEBUG: Upload Timeout: result=483328

...............................
DEBUG: Upload Stop:1613343279

DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 524288, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 1048576, 974848, 483328, 458752, 114688, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

DEBUG: Bytes sent:26148864

DEBUG: Upload Value:20746227

Upload: 20.75 Mbit/s
DEBUG: Results:
{'client': {'rating': '0', 'loggedin': '0', 'isprating': '3.7', 'ispdlavg': '0', 'ip': '193.116.x.x', 'isp': 'TPG Internet', 'lon': '153.0215', 'ispulavg': '0', 'country': 'AU', 'lat': '-27.4732'}, 'bytes_sent': 26148864, 'download': 0, 'timestamp': '2021-02-14T22:54:27.902614Z', 'share': None, 'bytes_received': 0, 'ping': 14.774, 'upload': 20746227.40552687, 'server': {'latency': 14.774, 'name': 'Brisbane', 'url': 'http://qld.speedtest.exetel.com.au:8080/speedtest/upload.php', 'country': 'Australia', 'lon': '153.0251', 'cc': 'AU', 'host': 'qld.speedtest.exetel.com.au:8080', 'sponsor': 'Exetel', 'lat': '-27.4698', 'id': '13276', 'd': 0.5187222967744429}}

###########################################################
# Python 3 Run
###########################################################

.DEBUG: Upload Start:224944

..DEBUG: Upload Timeout: result=524288

..DEBUG: Upload Timeout: result=524288

..DEBUG: Upload Timeout: result=524288

..DEBUG: Upload Timeout: result=524288

.............................................................................................
DEBUG: Upload Stop:224955

DEBUG: Bytes sent:[524288, 524288, 524288, 524288, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

DEBUG: Bytes sent:2097152

DEBUG: Upload Value:1574428

Upload: 1.57 Mbit/s
DEBUG: Results:
{'download': 0, 'upload': 1574428.9733984023, 'ping': 13.135, 'server': {'url': 'http://brs1.speedtest.telstra.net:8080/speedtest/upload.php', 'lat': '-27.4728', 'lon': '153.0278', 'name': 'Brisbane', 'country': 'Australia', 'cc': 'AU', 'sponsor': 'Telstra', 'id': '2604', 'host': 'brs1.speedtest.telstra.net:8080', 'd': 0.62311775977947, 'latency': 13.135}, 'timestamp': '2021-02-14T22:55:17.200248Z', 'bytes_sent': 2097152, 'bytes_received': 0, 'share': None, 'client': {'ip': '193.116.x.x', 'lat': '-27.4732', 'lon': '153.0215', 'isp': 'TPG Internet', 'isprating': '3.7', 'rating': '0', 'ispdlavg': '0', 'ispulavg': '0', 'loggedin': '0', 'country': 'AU'}}
[root@primary bin]#

 

 

I've temporarily resolved by forcing python2 in inputs.conf

 

[script://$SPLUNK_HOME/etc/apps/speedtest/bin/speedtest-cli.py --json]
disabled = 0
interval = 5,35 * * * *
python.version = python2
0 Karma

cameronjust
Path Finder

Thanks Mark. Happy to run some more tests if you need me to.

It's pretty odd though I have no idea how you would even begin to troubleshoot it or how it would behave like this only when run by Splunk.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...