Splunk Otel Chart Values.yaml for sending Otel Gateway logs to Splunk

[PhuN]

how do we enable Otel gateway logs to flow through to Splunk.

Even when we use the values.yaml settings noted here, we don't see any logs from the gateway: 

https://github.com/signalfx/splunk-otel-collector-chart/blob/main/examples/collector-gateway-only/co...

We're looking to get the gateway logs to get a better understanding of the health of the gateway 

[maulikp]

You need to include agent logs, and that will collect daemonset, clusterReceiver and gateway logs, you can configure it using this option - https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/...

[PhuN]

can you provide an example values.yaml with what you have in mind? 

Are you saying it should be this code below? if so, this yaml seems to output gateway logs but its not getting picked up and sent through to splunk.

clusterName: CHANGEME
splunkObservability:
  realm: CHANGEME
  accessToken: CHANGEME

gateway:
  enabled: true
  replicaCount: 1
  resources:
    limits:
      cpu: 2
      memory: 4Gi
agent:
  enabled: false
clusterReceiver:
  enabled: false
logsCollection:
  containers:
    excludeAgentLogs: false

[maulikp]

Sorry, I don't have a values.yaml file tailored for your use case. The values.yaml file you shared will only to setup a gateway on K8s cluster, which will not solve the problem you are looking to solve.

Let me ask you this, have you already installed Splunk OTEL helm chart on your K8s cluster, which is already running Daemonset and collecting logs?

[PhuN]

Yes the Splunk Otel helm chart is already on our K8 cluster and already collecting logs from all agents 

[maulikp · Accepted solution]

Great! In that case, you can update the existing values.yaml file with following, and redeploy the helm chart:

• In order to enable the gateway, set enabled to true in gateway section and adjust replicaCount and other configurations associated with gateway - https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/...
• Enable the agent logs via - https://github.com/signalfx/splunk-otel-collector-chart/blob/main/helm-charts/splunk-otel-collector/...

Once you redeploy your helm chart with above changes,

• You will have gateway running as part of helm chart
• OTEL agents logs will be collected via daemonset and sent to your backend.

[PhuN]

@maulikp Thanks. We're able to confirm gateway logs are now flowing through splunk now by searching for pod names that contain the word gateway
k8s.pod.name=*gateway* 

thank you very much

Phu