All Apps and Add-ons

Missing logs - Splunk Connect for Kubernetes

alanzchan
Path Finder

I have Splunk Connect for Kubernetes deployed in my GCP Kubernetes clusters. I see an issue where some of my container logs are not being indexed in my Splunk deployment. Looking at the GCP Stackdriver logs, I see all the container logs there.

Looking at the HTTP Event Collector logs:

{"datetime":"08-09-2019 08:45:30.806 -0400","log_level":"INFO","component":"HttpEventCollector","data":{"series":"http_event_collector","transport":"http","format":"json","total_bytes_received":7730368,"total_bytes_indexed":191166,"num_of_requests":105,"num_of_events":835,"num_of_errors":0,"num_of_parser_errors":105,"num_of_auth_failures":0,"num_of_requests_to_disabled_token":0,"num_of_requests_to_incorrect_url":0,"num_of_requests_in_mint_format":0,"num_of_ack_requests":0,"num_of_requests_acked":0,"num_of_requests_waiting_ack":0}}
{"datetime":"08-09-2019 08:45:30.806 -0400","log_level":"INFO","component":"HttpEventCollector","data":{"token_name":"k8s_token","series":"http_event_collector_token","transport":"http","format":"json","total_bytes_received":7730368,"total_bytes_indexed":191166,"num_of_requests":105,"num_of_events":835,"num_of_errors":0,"num_of_parser_errors":105,"num_of_requests_to_disabled_token":0,"num_of_requests_in_mint_format":0}}

I see a number of "num_of_parser_errors". I also see that the total number of bytes received is significantly smaller than the total number of bytes indexed.

Any ideas?

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...