All Apps and Add-ons

Why is splunk-connect-for-kubernetes search by namespace=mynamespace missing results?

fazie
New Member

Hi,
We use splunk-connect-for-kubernetes to send logs to splunk via HEC mechanism. Sending logs to splunk is fine, but searching is not.
When we search for

namespace=mynamespace "*Exception*"

There is lots of missing logs, very few is returned.
But, when I search like that:

namespace=*mynamespace* "*Exception*"

All is fine, all logs are returned

Any suggestions?

OUtput part of fluentd configuration:

  <match **>
    @type copy
    deep_copy true
    <store>
      @type splunk_hec
      protocol https
      hec_host "#{ENV['SPLUNK_HOST']}"
      hec_port "#{ENV['SPLUNK_PORT']}"
      hec_token "#{ENV['SPLUNK_TOKEN']}"
      host "#{ENV['NODE_NAME']}"
      source_key source
      sourcetype_key sourcetype
      <fields>
        pod
        namespace
        container_name
        container_id
        cluster_env
        cluster_name
      </fields>
      <buffer>
        @type memory
        chunk_limit_records 100000
        chunk_limit_size 200m
        flush_interval 5s
        flush_thread_count 1
        overflow_action block
        retry_max_times 3
        total_limit_size 600m
      </buffer>
      <format>
        @type single_value
        message_key log
        add_newline false
      </format>
    </store>
    <store>
      @type prometheus
      <metric>
        (...)
      </metric>
    </store>
  </match>
Labels (1)
0 Karma

RDumbeck
Explorer

Try using namespace::mynamespace

I cannot remember why though.

0 Karma

RDumbeck
Explorer

any luck with this. I have the same problem.

0 Karma

efloss
Engager

Ran into this recently, putting them together in a fields.conf file on the search head will make them searchable without needing the wildcards since they're metadata fields.

[k8s.cluster.name]
INDEXED=true

[k8s.container.name]
INDEXED=true

[k8s.namespace.name]
INDEXED=true

[k8s.node.name]
INDEXED=true

[k8s.pod.name]
INDEXED=true

[k8s.pod.uid]
INDEXED=true

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...