topic Re: Number of appserver.py processes increasing, causing OOM in Monitoring Splunk

Number of appserver.py processes increasing, causing OOM

hrawat — Wed, 14 Feb 2024 23:50:46 GMT

Search Head appears to have a rogue python process ( appserver.py) that slowly eats away all memory on the system, then eventually causes an OOM, which requires a manual restart of splunkd, then the issue starts slowly creeping up to happen again.

Re: Number of appserver.py processes increasing, causing OOM

hrawat — Wed, 14 Feb 2024 23:51:43 GMT

Due to some issue with proper cleanup of idle processes, number of python process ( appserver.py) running on the system constantly grow. Thus due to systemwide memory growth, these stale processes, eventually causes an OOM.

Run following search to find if any search head is impacted by this issue and what % of total system memory these stale processes running more than 24 hours. If these processes using more than 15% of total system memory, then run script to kill stales processes.

index=_introspection host=<all search heads> appserver.py data.elapsed > 86400 | dedup host, data.pid | stats dc(data.pid) as cnt sum("data.pct_memory") AS appserver_memory_used by host | sort - appserver_memory_used

On linux/unix you can use following script to kill stale processes and reclaim memory.

kill -TERM $(ps -eo etimes,pid,cmd | awk '{if ( $1 >= 86400) print $2 " " $4 }' |grep appserver.py | awk '{print $1}')

Re: Number of appserver.py processes increasing, causing OOM

waechtler_amaso — Tue, 30 Jul 2024 09:53:44 GMT

I see this behaviour, too, also for another process coming from the ITSI app:

/opt/splunk/etc/apps/SA-ITOA/bin/command_health_monitor.py

Besides killing processes or restarting splunk as a workaround, do you know whether there are efforts to finally resolve this bug?

Thanks, Jan

Re: Number of appserver.py processes increasing, causing OOM

hrawat — Tue, 30 Jul 2024 10:23:01 GMT

Splunk 9.3.0 has the fix.