All Apps and Add-ons

Splunk AddOn Microsoft Cloudservices Memory Leak?

gazoscreek
Loves-to-Learn Lots

We're seeing severe memory issues with the Splunk AddOn for Microsoft Cloudservices where a single python process can consume well over 200Gb of memory, ultimately being reaped by the OOM Killer.

These are Azure Firewall collections. Rather critical to our logging infrastructure and we have no information as to why this is happening. It doesn't happen to all collections, but does happen to many ... You can see below, one child process consuming > 60Gb ... and Splunkd having grown to 275Gb of virtual memory.

splunk 17311 125 67318812 62146568 /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_storage_blob.py
splunk 15910 227 274447324 82630100 splunkd -p 8089 restart

Splunk version 8.0.4.1

Splunk_TA_microsoft-cloudservices version 4.1.2

 

 

Labels (2)
0 Karma

dodland
Engager

Did you ever get any traction on this? I have the same exact issue on two servers. One has 16GB of memory, the other has 56GB. Both seem to hover around the 68% mark for RAM usage.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!