Splunk Search

Inconsistent results when searching splunk connect for kubernetes logs

schose
Builder

Hi all,

I’m getting strange results when splunking container logs collected by splunk connect for k8s…

when searching for the pod logs with normal SPL:

 

 

 

index=kubernetes earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00 pod=nextcloud-dev-84ff5f7dfb-jj2gz | stats count

 

 

 

result=0


when forcing running the stats on shd:

 

 

 

index=kubernetes earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00 | noop | search pod=nextcloud-dev-84ff5f7dfb-jj2gz | stats count

 

 

 

result=131 - looks correct

when running tstats:

 

 

 

| tstats count where index=kubernetes pod=nextcloud-dev-84ff5f7dfb-jj2gz earliest=08/24/2020:17:00:00 latest=08/24/2020:17:45:00

 

 

 

result=131 - looks correct…

 

What may be the issue for the "normal" search noch working for the user?

Running Splunk Enterprise v8.0.5.

 

best regards,

 

Andreas

Labels (2)
Tags (2)
0 Karma

schose
Builder

hi all,

As pod is an indexed field you have to configure fields.conf or search bei pod::mypod.. 

it's stated in the documentation:

 

"Splunk Connect for Kubernetes sends events to Splunk which can contain extra meta-data attached to each event. Metadata values such as "pod", "namespace", "container_name","container_id", "cluster_name" will appear as fields when viewing the event data inside Splunk. There are two solutions for running searches in Splunk on meta-data.

  • Modify search to usefieldname::value instead of fieldname=value.
  • Configure fields.conf on your downstream Splunk system to have your meta-data fields available to be searched using fieldname=value. Example: fields.conf.example

"

fields.conf example:
https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/fields.conf.example

 

sorry,

 

Andreas

0 Karma
Get Updates on the Splunk Community!

Splunk at Cisco Live 2025: Learning, Innovation, and a Little Bit of Mr. Brightside

Pack your bags (and maybe your dancing shoes)—Cisco Live is heading to San Diego, June 8–12, 2025, and Splunk ...

Splunk App Dev Community Updates – What’s New and What’s Next

Welcome to your go-to roundup of everything happening in the Splunk App Dev Community! Whether you're building ...

The Latest Cisco Integrations With Splunk Platform!

Join us for an exciting tech talk where we’ll explore the latest integrations in Cisco + Splunk! We’ve ...