I have Splunk Connect for Kubernetes deployed in my GCP Kubernetes clusters. I see an issue where some of my container logs are not being indexed in my Splunk deployment. Looking at the GCP Stackdriver logs, I see all the container logs there.
Looking at the HTTP Event Collector logs:
{"datetime":"08-09-2019 08:45:30.806 -0400","log_level":"INFO","component":"HttpEventCollector","data":{"series":"http_event_collector","transport":"http","format":"json","total_bytes_received":7730368,"total_bytes_indexed":191166,"num_of_requests":105,"num_of_events":835,"num_of_errors":0,"num_of_parser_errors":105,"num_of_auth_failures":0,"num_of_requests_to_disabled_token":0,"num_of_requests_to_incorrect_url":0,"num_of_requests_in_mint_format":0,"num_of_ack_requests":0,"num_of_requests_acked":0,"num_of_requests_waiting_ack":0}}
{"datetime":"08-09-2019 08:45:30.806 -0400","log_level":"INFO","component":"HttpEventCollector","data":{"token_name":"k8s_token","series":"http_event_collector_token","transport":"http","format":"json","total_bytes_received":7730368,"total_bytes_indexed":191166,"num_of_requests":105,"num_of_events":835,"num_of_errors":0,"num_of_parser_errors":105,"num_of_requests_to_disabled_token":0,"num_of_requests_in_mint_format":0}}
I see a number of "num_of_parser_errors". I also see that the total number of bytes received is significantly smaller than the total number of bytes indexed.
Any ideas?
