Amazon Linux 2023 performance troubleshooting

[briancronrath]

I have been tasked with building out new instances of anything that runs an older OS, and for our EC2 instances this most often means I have to rebuild the instance on Amazon Linux 2023.  What I've noticed, particularly with our heavy forwarders, is that they are constantly dying and ballooning memory to 100% with no rhyme or reason as to why (not heavy load, happens to random servers at random times, and there should be no world where the memory even comes close to getting consumed considering the low level of traffic for the servers in question which are in our lab environment).  

We're running splunk version 10.0.2

Nothing helpful ever in the logs aside from indications that it ran OOM.  I've randomly tested with making absurdly large instances to see if the issue still occurs and sure enough it does, there's not enough memory in the world to handle whatever error seems to be occurring, so it seems to be something more akin to a bug that's occurring.

I'm curious if anyone else has ran into issues with AL2023 and running Splunk?

[PickleRick]

This is not very probable that it's your case but I had such behaviour (and it was on HFs as in your case) when the KV-store was not fully upgraded.

For some reason the HFs didn't fully migrate from the previous engine to wiredtiger. The files on the disk were still from the pre-wiredtiger mongo but the configuration suggested it was wiredtiger. As a result, the kvstore would start but after a very short time would die. And splunkd would start pumping up memory usage up to the point when OOM would kill it. Then the service would get automatically restarted and fun would start again.

We had a "cluster" of 4 intermediate HF and they were restarting one at a time so we actually found out about it kinda by accident.

But it was 9.0. I'm not sure if your 10.0.2 would even try to start not having fully migrated to wiredtiger.