Splunk SOAR (f.k.a. Phantom)

Ingest daemon troubleshooting: Where to look for the root cause?

BorkoG
Engager

Hi folks,

Our on-premise 5.3.1 SOAR's Ingest daemon is behaving funny in terms of memory management and was wondering if someone can give me any pointers to where to look for what is going wrong.

In essence, the ingestd keeps on using more and more virtual memory until it maxes out at 256GB and then stops ingesting more data. Restarting the service does solve the issue.

BorkoG_0-1674752788902.png

I am thinking the root cause might be hiding in 3 places:
- poorly written playbooks - I am thinking something might be wrong with the playbooks that we have. We have playbooks running as often as every 5 minutes, so I suppose they can cause resource starvation. Not sure how to dive deeper for potential memory leaks here though. 

- something going wrong with the ingestion of containers/better clean-up of closed containers - is it possible that just closing containers without deleting them after X amount of time can cause this?

- some weird bug that we've hit - not sure how likely this is but I saw that in version 5.3.4 a bug regarding memory usage has been fixed (PSAAS-9663) so it is on my list, if nothing else turns up

 

One relevant point to make is that this started occurring after migration from 4.9.X to our current version so I have no idea if this is linked to the fact that we migrated to Python 3 playbooks or the particular product version.

Any pointers to where/how to start looking for the root cause are appreciated.

Cheers.

Labels (2)
Tags (2)
0 Karma
1 Solution

BorkoG
Engager

So this turned out to be the PSAAS-8617 issue in 5.3.1. The only solution is to update to the 5.3.2 or later version.

View solution in original post

0 Karma

BorkoG
Engager

So this turned out to be the PSAAS-8617 issue in 5.3.1. The only solution is to update to the 5.3.2 or later version.

0 Karma
Get Updates on the Splunk Community!

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...