Why is pgrade Failing with OSError type 28?

edwardrose — Mon, 28 Aug 2023 18:12:31 GMT

Hello All,

I am testing the upgrade from ES 6.2.0 to 6.6.2. When I do the upgrade it fails with OSError type 28 no space left of device. But there is almost 30GB of disk space free.

2021-10-04 19:18:28,028 INFO [615bb5deed7f2dc4595650] _cplogging:216 - [04/Oct/2021:19:18:28] HTTP Request Headers: Remote-Addr: 127.0.0.1 TE: chunked HOST: splunk-sh1.wv.mentorg.com:8000 ACCEPT-ENCODING: gzip, br CACHE-CONTROL: max-age=0 SEC-CH-UA: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93" SEC-CH-UA-MOBILE: ?0 SEC-CH-UA-PLATFORM: "Windows" UPGRADE-INSECURE-REQUESTS: 1 ORIGIN: null USER-AGENT: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36 ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 SEC-FETCH-SITE: same-origin SEC-FETCH-MODE: navigate SEC-FETCH-USER: ?1 SEC-FETCH-DEST: document ACCEPT-LANGUAGE: en-US,en;q=0.9 COOKIE: splunkweb_csrf_token_8000=[REDACTED]5649; session_id_8000=[REDACTED]5b74; token_key=[REDACTED]5649; experience_id=[REDACTED]b0c2; splunkd_8000=[REDACTED]tgchx REMOTE-USER: admin X-SPLUNKD: SKdIpkhtf8PlfUDwvOLunA== 11626949294704615649 ijbs1HY^4Ms541EE5sF6eqHg^iyD5t6QKZRByWhdMDXkj546^eB1lT6y59b9LewgHbLcz0Xa5SKotHijcl__zWhYqh8MZISrCqYVxuLkY7jijwyyXijSUQ9VAJRlcQA3o7tgchx 0 Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryO0HdVIPxgJr5HUZN Content-Length: 675766277 2021-10-04 19:18:28,029 INFO [615bb5deed7f2dc4595650] error:333 - POST /en-US/manager/appinstall/_upload 127.0.0.1 8065 2021-10-04 19:18:28,029 INFO [615bb5deed7f2dc4595650] error:334 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request. 2021-10-04 19:18:28,029 ERROR [615bb5deed7f2dc4595650] error:335 - Traceback (most recent call last): File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond self._do_respond(path_info) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 680, in _do_respond self.body.process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 982, in process super(RequestBody, self).process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 559, in process proc(self) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 225, in process_multipart_form_data process_multipart(entity) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 217, in process_multipart part.process() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 557, in process self.default_proc() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 717, in default_proc self.file = self.read_into_file() File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 732, in read_into_file self.read_lines_to_boundary(fp_out=fp_out) File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 702, in read_lines_to_boundary fp_out.write(line) OSError: [Errno 28] No space left on device

As you can see there should be plenty of room for a 670MB upload

splunk@splunk-sh1:~/var/log/splunk> df -kh /opt/splunk Filesystem Size Used Avail Use% Mounted on /dev/mapper/system-splunk 74G 44G 27G 63% /opt splunk@splunk-sh1:~/var/log/splunk>

Web.conf

splunk@splunk-sh1:~/var/log/splunk> more ~/etc/system/local/web.conf [settings] login_content = <h1> <CENTER>Splunk Dev Search Head</CENTER> </h1> max_upload_size = 1024 enableSplunkWebSSL = 1 privKeyPath = /opt/splunk/etc/auth/splunkweb/com.key caCertPath = /opt/splunk/etc/auth/splunkweb/expJun2022.crt splunkdConnectionTimeout = 1400 tools.sessions.timeout = 180 sslVersions = ssl3,tls cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH splunk@splunk-sh1:~/var/log/splunk>

So I am confused why it would say that there is no space left of the device.

Thanks

Re: Upgrade Failing with OSError type 28

edwardrose — Tue, 05 Oct 2021 12:51:12 GMT

I found this article

https://community.splunk.com/t5/Getting-Data-In/Tutorial-data-upload-error/m-p/214891

I reduce the setting as stated to 4800 and the upgrade proceeded just fine.

Weird.

Re: Upgrade Failing with OSError type 28

smithy001 — Thu, 17 Feb 2022 14:42:46 GMT

Had the same issue.

Cleared /tmp down and worked fine...

Can only assume splunk GUI loads to /tmp

Re: Upgrade Failing with OSError type 28

computermathguy — Mon, 28 Aug 2023 16:38:24 GMT

We managed to resolve the the "type 28 / 500 internal server" Enterprise Security installation error by cleaning out /tmp.

topic Why is pgrade Failing with OSError type 28? in Splunk Enterprise Security

Why is pgrade Failing with OSError type 28?

Re: Upgrade Failing with OSError type 28

Re: Upgrade Failing with OSError type 28

Re: Upgrade Failing with OSError type 28