Installation

Install Python for Scientific Computing on cluster - tarball too large and caused Network-layer error: Broken pipe

patng_nw
Communicator

I tried to upgrade Python for Scientific Computing to v5.3 on my cluster.  I followed the instructions in here and first un-tar the add-on on my master node.

However, whenever I ran /opt/splunk/bin/splunk apply shcluster-bundle, I will always get this error.

Error while deploying apps to first member, aborting apps deployment to all members: 
Error while updating app=Splunk_SA_Scientific_Python_linux_x86_64 on
target=https://172.18.109.2:8089: Network-layer error: Broken pipe

This kind of error didn't happen when I installed another smaller add-on earlier this morning.  And after failing with that error multiple times, I noticed the doc says:

If you attempt to push a very large tarball (>200 MB), the operation might fail due to various timeouts. Delete some of the contents from the tarball's app, if possible, and try again.

Since the original python-for-scientific-computing-for-linux-64-bit_300.tgz is already 480MB, and the size of the un-tar and  unzipped /opt/splunk/etc/shcluster/apps/Splunk_SA_Scientific_Python_linux_x86_64 folder on the master node is 2.5G!  I bet the large tarball size is the problem.

But how can I solve this problem?  What contents from the tarball's app can I delete?

Labels (3)
1 Solution

patng_nw
Communicator
I have resolved the problem myself.  
 
On the deployer, the size of the folder /opt/splunk/etc/shcluster/apps/Splunk_SA_Scientific_Python_linux_x86_64 kept growing because several older versions of "Python for Scientific Computing" were deployed in the past (which is normal) and the libraries for Python 2.7, 3.7 and 3.8 (the latest one) were all inside, resulting in the folder's total size becoming like 5GB.  
 
The solution is to remove the Splunk_SA_Scientific_Python_linux_x86_64 folder, then un-tar the python-for-scientific-computing-for-linux-64-bit_300.tgz file, and then masterBundlePush.sh won't fail this time.
 

View solution in original post

patng_nw
Communicator
I have resolved the problem myself.  
 
On the deployer, the size of the folder /opt/splunk/etc/shcluster/apps/Splunk_SA_Scientific_Python_linux_x86_64 kept growing because several older versions of "Python for Scientific Computing" were deployed in the past (which is normal) and the libraries for Python 2.7, 3.7 and 3.8 (the latest one) were all inside, resulting in the folder's total size becoming like 5GB.  
 
The solution is to remove the Splunk_SA_Scientific_Python_linux_x86_64 folder, then un-tar the python-for-scientific-computing-for-linux-64-bit_300.tgz file, and then masterBundlePush.sh won't fail this time.
 
Get Updates on the Splunk Community!

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

See your relevant APM services, dashboards, and alerts in one place with the updated ...

As a Splunk Observability user, you have a lot of data you have to manage, prioritize, and troubleshoot on a ...