Security

Splunkweb consumes large amounts of memory and appears to hang

Splunk Employee
Splunk Employee

On restart splunkweb starts up, but does not respond to attempts to connect via web. Nestat showed that we were listening on the port.

Over the course of half an hour splunkweb gradually consumed all memory on the box (24 Gb).

If you attempted to restart - splunkweb wouldn't respond before the timeout and would therefore be killed.

Splunk was version 4.3.1.

Tags (1)
1 Solution

Splunk Employee
Splunk Employee

Turned out this was caused by a known bug SPL-48237.

The problem is that Splunk doesn't always clear up the session lock files found within {splunk install}/var/run/splunk. We had many thousands of these session lock files that SplunkWeb was trying to process.

Deleting the files manually allowed splunkweb to restart. Fixed in 4.3.4 and later.

View solution in original post

Splunk Employee
Splunk Employee

Turned out this was caused by a known bug SPL-48237.

The problem is that Splunk doesn't always clear up the session lock files found within {splunk install}/var/run/splunk. We had many thousands of these session lock files that SplunkWeb was trying to process.

Deleting the files manually allowed splunkweb to restart. Fixed in 4.3.4 and later.

View solution in original post