Splunk SOAR

Ingest daemon troubleshooting: Where to look for the root cause?

BorkoG
Engager

Hi folks,

Our on-premise 5.3.1 SOAR's Ingest daemon is behaving funny in terms of memory management and was wondering if someone can give me any pointers to where to look for what is going wrong.

In essence, the ingestd keeps on using more and more virtual memory until it maxes out at 256GB and then stops ingesting more data. Restarting the service does solve the issue.

BorkoG_0-1674752788902.png

I am thinking the root cause might be hiding in 3 places:
- poorly written playbooks - I am thinking something might be wrong with the playbooks that we have. We have playbooks running as often as every 5 minutes, so I suppose they can cause resource starvation. Not sure how to dive deeper for potential memory leaks here though. 

- something going wrong with the ingestion of containers/better clean-up of closed containers - is it possible that just closing containers without deleting them after X amount of time can cause this?

- some weird bug that we've hit - not sure how likely this is but I saw that in version 5.3.4 a bug regarding memory usage has been fixed (PSAAS-9663) so it is on my list, if nothing else turns up

 

One relevant point to make is that this started occurring after migration from 4.9.X to our current version so I have no idea if this is linked to the fact that we migrated to Python 3 playbooks or the particular product version.

Any pointers to where/how to start looking for the root cause are appreciated.

Cheers.

Labels (2)
Tags (2)
0 Karma
1 Solution

BorkoG
Engager

So this turned out to be the PSAAS-8617 issue in 5.3.1. The only solution is to update to the 5.3.2 or later version.

View solution in original post

0 Karma

BorkoG
Engager

So this turned out to be the PSAAS-8617 issue in 5.3.1. The only solution is to update to the 5.3.2 or later version.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...