Sending logs from kubernetes to on prem Splunk Enterprise?

[nmsvuuk]

Hi

Trying to figure out how best to send the logs (at least kubernetes logs+ possibly application logs) from on prem Kubernetes cluster to on prem Splunk Enterprise.

I have gone through a long list of options such as 'Splunk app for infrastructure' (EOL),  Splunk connect for kubernetes ( EOL Jan 2024) ,  Splunk Operators v1 and 2  etc

Splunk OpenTelemetry Collector for Kubernetes would look promising but if I understood correctly this only works with observatility (cloud) and not meant to work/ supported  with with Splunk Enterprise.

My question is what is the best way to ship logs from Splunk to Splunk enterprise (both on prem) ? Currently the Logging , Metrics , Traces etc have not yet been configured on Kubernetes clluster I am building. Since we use Splunk for certalized log collection, what ever solution it is needs to work with Splunk enterprise but also ideally be more futureproof than many of solutions seen previously.

[durga-hp]

 Hello Team,

We are seeing some weirdness when are sending logs to splunk enterprise on-prem. Prior we used to use splunk OTEL Java agent V1 and things were fine, Once the migration to Splunk OTEL Java agent V2 was done, we started seeing logs being duplicated like below

The below is what started showing up

Can you please help how to we stop the kubernetes source?

The actual source which we used to observe is as below

Please let me know if you need any more information. I would really appreciate any insights into this and arresting the logs from source kubernetes. Thanks!

[Gr0und_Z3r0]

hi @nmsvuuk 

You can do so by installing the Splunk Universal forwarder on the master of the cluster and configure the /var/log path for cluster master logs and  for any application related logs, you can use HTTP Event Collector to ship logs to Splunk. 

[nmsvuuk]

Hi

I was first thinking there must be a solution to be deployed directly to the Kubernetes as deployment/pods but if installing standard Splunk forwarder on OS level and shipping logs to Splunk enterprise  is a good solution for Kubernetes logs, I  may just try that out . 

I will have a look at the HTTPS collector to see how well it addresses the application side logging.

Thanks for your help!