Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe

patng_nw — Tue, 09 Nov 2021 07:13:39 GMT

I tried to upgrade Python for Scientific Computing to v5.3 on my cluster. I followed the instructions in here and first un-tar the add-on on my master node.

However, whenever I ran /opt/splunk/bin/splunk apply shcluster-bundle, I will always get this error.

Error while deploying apps to first member, aborting apps deployment to all members: 
Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on 
target=https://172.18.109.2:8089: Network-layer error: Broken pipe

This kind of error didn't happen when I installed another smaller add-on earlier this morning. And after failing with that error multiple times, I noticed the doc says:

If you attempt to push a very large tarball (>200 MB), the operation might fail due to various timeouts. Delete some of the contents from the tarball's app, if possible, and try again.

Since the original python-for-scientific-computing-for-linux-64-bit_300.tgz is already 480MB, and the size of the un-tar and unzipped /opt/splunk/etc/shcluster/apps/Splunk_SA_Scientific_Python_linux_x86_64 folder on the master node is 2.5G! I bet the large tarball size is the problem.

But how can I solve this problem? What contents from the tarball's app can I delete?

Re: Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe

patng_nw — Wed, 17 Nov 2021 07:45:25 GMT

I have resolved the problem myself.

On the deployer, the size of the folder /opt/splunk/etc/shcluster/apps/Splunk_SA_Scientific_Python_linux_x86_64 kept growing because several older versions of "Python for Scientific Computing" were deployed in the past (which is normal) and the libraries for Python 2.7, 3.7 and 3.8 (the latest one) were all inside, resulting in the folder's total size becoming like 5GB.

The solution is to remove the Splunk_SA_Scientific_Python_linux_x86_64 folder, then un-tar the python-for-scientific-computing-for-linux-64-bit_300.tgz file, and then masterBundlePush.sh won't fail this time.

topic Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe in Installation

Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe

Re: Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe