topic Re: Why is splunk-connect-for-kubernetes search by namespace=mynamespace missing results? in All Apps and Add-ons

Why is splunk-connect-for-kubernetes search by namespace=mynamespace missing results?

fazie — Thu, 01 Dec 2022 16:16:14 GMT

Hi,
We use splunk-connect-for-kubernetes to send logs to splunk via HEC mechanism. Sending logs to splunk is fine, but searching is not.
When we search for

namespace=mynamespace "*Exception*"

There is lots of missing logs, very few is returned.
But, when I search like that:

namespace=*mynamespace* "*Exception*"

All is fine, all logs are returned

Any suggestions?

OUtput part of fluentd configuration:

  <match **>
    @type copy
    deep_copy true
    <store>
      @type splunk_hec
      protocol https
      hec_host "#{ENV['SPLUNK_HOST']}"
      hec_port "#{ENV['SPLUNK_PORT']}"
      hec_token "#{ENV['SPLUNK_TOKEN']}"
      host "#{ENV['NODE_NAME']}"
      source_key source
      sourcetype_key sourcetype
      <fields>
        pod
        namespace
        container_name
        container_id
        cluster_env
        cluster_name
      </fields>
      <buffer>
        @type memory
        chunk_limit_records 100000
        chunk_limit_size 200m
        flush_interval 5s
        flush_thread_count 1
        overflow_action block
        retry_max_times 3
        total_limit_size 600m
      </buffer>
      <format>
        @type single_value
        message_key log
        add_newline false
      </format>
    </store>
    <store>
      @type prometheus
      <metric>
        (...)
      </metric>
    </store>
  </match>

Re: splunk-connect-for-kubernetes search by namespace=mynamespace missing results

RDumbeck — Wed, 30 Nov 2022 18:17:08 GMT

any luck with this. I have the same problem.

Re: splunk-connect-for-kubernetes search by namespace=mynamespace missing results

efloss — Mon, 12 Dec 2022 15:54:16 GMT

Ran into this recently, putting them together in a fields.conf file on the search head will make them searchable without needing the wildcards since they're metadata fields.

[k8s.cluster.name]
INDEXED=true

[k8s.container.name]
INDEXED=true

[k8s.namespace.name]
INDEXED=true

[k8s.node.name]
INDEXED=true

[k8s.pod.name]
INDEXED=true

[k8s.pod.uid]
INDEXED=true

Re: Why is splunk-connect-for-kubernetes search by namespace=mynamespace missing results?

RDumbeck — Wed, 03 May 2023 20:06:00 GMT

Try using namespace::mynamespace

I cannot remember why though.