Splunk Enterprise Security

Why is pgrade Failing with OSError type 28?

edwardrose
Contributor

Hello All,

I am testing the upgrade from ES 6.2.0 to 6.6.2.  When I do the upgrade it fails with OSError type 28 no space left of device.  But there is almost 30GB of disk space free.

 

 

2021-10-04 19:18:28,028 INFO    [615bb5deed7f2dc4595650] _cplogging:216 - [04/Oct/2021:19:18:28] HTTP
Request Headers:
  Remote-Addr: 127.0.0.1
  TE: chunked
  HOST: splunk-sh1.wv.mentorg.com:8000
  ACCEPT-ENCODING: gzip, br
  CACHE-CONTROL: max-age=0
  SEC-CH-UA: "Google Chrome";v="93", " Not;A Brand";v="99", "Chromium";v="93"
  SEC-CH-UA-MOBILE: ?0
  SEC-CH-UA-PLATFORM: "Windows"
  UPGRADE-INSECURE-REQUESTS: 1
  ORIGIN: null
  USER-AGENT: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
  SEC-FETCH-SITE: same-origin
  SEC-FETCH-MODE: navigate
  SEC-FETCH-USER: ?1
  SEC-FETCH-DEST: document
  ACCEPT-LANGUAGE: en-US,en;q=0.9
  COOKIE: splunkweb_csrf_token_8000=[REDACTED]5649; session_id_8000=[REDACTED]5b74; token_key=[REDACTED]5649; experience_id=[REDACTED]b0c2; splunkd_8000=[REDACTED]tgchx
  REMOTE-USER: admin
  X-SPLUNKD: SKdIpkhtf8PlfUDwvOLunA== 11626949294704615649 ijbs1HY^4Ms541EE5sF6eqHg^iyD5t6QKZRByWhdMDXkj546^eB1lT6y59b9LewgHbLcz0Xa5SKotHijcl__zWhYqh8MZISrCqYVxuLkY7jijwyyXijSUQ9VAJRlcQA3o7tgchx 0
  Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryO0HdVIPxgJr5HUZN
  Content-Length: 675766277
2021-10-04 19:18:28,029 INFO    [615bb5deed7f2dc4595650] error:333 - POST /en-US/manager/appinstall/_upload 127.0.0.1 8065
2021-10-04 19:18:28,029 INFO    [615bb5deed7f2dc4595650] error:334 - 500 Internal Server Error The server encountered an unexpected condition which prevented it from fulfilling the request.
2021-10-04 19:18:28,029 ERROR   [615bb5deed7f2dc4595650] error:335 - Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 628, in respond
    self._do_respond(path_info)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cprequest.py", line 680, in _do_respond
    self.body.process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 982, in process
    super(RequestBody, self).process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 559, in process
    proc(self)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 225, in process_multipart_form_data
    process_multipart(entity)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 217, in process_multipart
    part.process()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 557, in process
    self.default_proc()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 717, in default_proc
    self.file = self.read_into_file()
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 732, in read_into_file
    self.read_lines_to_boundary(fp_out=fp_out)
  File "/opt/splunk/lib/python3.7/site-packages/cherrypy/_cpreqbody.py", line 702, in read_lines_to_boundary
    fp_out.write(line)
OSError: [Errno 28] No space left on device

 

 

 

 

 

As you can see there should be plenty of room for a 670MB upload

 

 

splunk@splunk-sh1:~/var/log/splunk> df -kh /opt/splunk
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/system-splunk   74G   44G   27G  63% /opt
splunk@splunk-sh1:~/var/log/splunk>

 

 

Web.conf

 

 

splunk@splunk-sh1:~/var/log/splunk> more ~/etc/system/local/web.conf
[settings]
login_content = <h1> <CENTER>Splunk Dev Search Head</CENTER> </h1>
max_upload_size = 1024
enableSplunkWebSSL = 1
privKeyPath = /opt/splunk/etc/auth/splunkweb/com.key
caCertPath = /opt/splunk/etc/auth/splunkweb/expJun2022.crt
splunkdConnectionTimeout = 1400
tools.sessions.timeout = 180
sslVersions = ssl3,tls
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

splunk@splunk-sh1:~/var/log/splunk>

 

 

So I am confused why it would say that there is no space left of the device.

Thanks

ed

Labels (2)
Tags (2)
0 Karma
1 Solution

edwardrose
Contributor

I found this article

 

https://community.splunk.com/t5/Getting-Data-In/Tutorial-data-upload-error/m-p/214891

 

I reduce the setting as stated to 4800 and the upgrade proceeded just fine.

Weird.

View solution in original post

0 Karma

edwardrose
Contributor

I found this article

 

https://community.splunk.com/t5/Getting-Data-In/Tutorial-data-upload-error/m-p/214891

 

I reduce the setting as stated to 4800 and the upgrade proceeded just fine.

Weird.

0 Karma

smithy001
Explorer

Had the same issue.

Cleared /tmp down and worked fine...

Can only assume splunk GUI loads to /tmp

Tags (1)
0 Karma

computermathguy
Explorer

We managed to resolve the the "type 28 / 500 internal server" Enterprise Security installation error by cleaning out /tmp.  

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...