504 Gateway Time-out

[Mohanad]

We are hosting Splunk enterprise on AWS EC2 instances, the flow goes as follows:

ALB>Apache Reverse proxies>ALB>SHC<>Indexers.

after a period of times (days mostly) we start to experience 504 gateway time-out which disappears when we restart our proxies, and we go for another round and so on.
Any clues for how to troubleshoot this,
we adjusted the timeouts parameters on the application, and the application loadbalancers but the problem is still persisting.

Mohanad_0-1730201872578.png

[dural_yyz]

Take a look at any TLS certificates that get issued between ALB and Proxy.