"Hipster Store Guided Onboarding Demo App K8s Pods Stuck in CrashLoopBackOff”

[jasonrogers]

Hi all,

I am deploying the Hipster Store application as part of the Guided Onboarding in the Splunk Observability Cloud. However, many of the pods are stuck in `CrashLoopBackOff` status.

Environment Details

- Platform: Minikube  

- Minikube Version: v1.34.0 (running on macOS 14.5, arm64)  

- Docker Version: 

  - Client: v27.4.0  

  - Server: v27.2.0 (Community Edition)  

- Helm Version: version.BuildInfo{Version:"v3.16.4", GitCommit:"7877b45b63f95635153b29a42c0c2f4273ec45ca", GitTreeState:"clean", GoVersion:"go1.22.7"}

Command Used for Deployment

helm install my-sixth-splunk-o11y-otel-collector --set="splunkObservability.realm=us1,splunkObservability.accessToken=XXX_XXXXXXXXXXXXXXXXXX,clusterName=my-sixth-otel-collector-cluster" splunk-otel-collector-chart/splunk-otel-collector

Error Observed 

Pods remain in the `CrashLoopBackOff` state. Below is the output of `kubectl get pods`:

NAME                                                              READY   STATUS             RESTARTS        AGE

adservice-cbbc87864-fbt7d                                         0/1     Running            3 (45s ago)     7m51s

cartservice-797fcdd44b-67ctq                                      0/1     CrashLoopBackOff   6 (97s ago)     7m52s

checkoutservice-7c5955d5b9-gksms                                  0/1     CrashLoopBackOff   6 (2m11s ago)   7m52s

... (other entries truncated for brevity)

Logs from Affected Pods

1. `checkoutservice` Logs:

   {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:58:09.648950291Z"}

   {"message":"sleeping 10s to retry initializing Stackdriver profiler","severity":"info"}

2. `productcatalogservice` Logs:

   {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:59:15.143462044Z"}

   {"message":"sleeping 20s to retry initializing Stackdriver profiler","severity":"info"}

3. `shippingservice` Logs:

   {"message":"failed to start profiler: project ID must be specified in the configuration if running outside of GCP","severity":"warning","timestamp":"2025-01-06T22:58:59.462097926Z"}

   {"message":"sleeping 10s to retry initializing Stackdriver profiler","severity":"info"}

Troubleshooting Attempts

1. Tried adding dummy Google Project IDs (I researched how they should formatted) in various configuration files.  

2. Disabled Tracing, Profiling, and Debugging at different times.  

3. Confirmed Helm chart installation was successful.  

Questions

1. How can I resolve the `CrashLoopBackOff` issue?  

2. Is there a way to bypass or properly configure the Stackdriver profiler to avoid requiring a Google Project ID?  

3. Are there any additional configurations required to run this demo outside of GCP?

Thank you.

[bishida]

I'm thinking the Hipster shop demo environment may be a lot of extra work to get running on MacOS/arm64.  In the following docs, they mention preferred option is to run in GKE. If you run locally, they did it with ubuntu Linux.

https://github.com/GoogleCloudPlatform/microservices-demo/blob/main/docs/development-guide.md

[jasonrogers]

Hi bishida, I appreciate your response. Thanks for pointer to the development guide. I'll give it a try.