All Apps and Add-ons

Why is splunk-connect-for-kubernetes search by namespace=mynamespace missing results?

fazie
New Member

Hi,
We use splunk-connect-for-kubernetes to send logs to splunk via HEC mechanism. Sending logs to splunk is fine, but searching is not.
When we search for

namespace=mynamespace "*Exception*"

There is lots of missing logs, very few is returned.
But, when I search like that:

namespace=*mynamespace* "*Exception*"

All is fine, all logs are returned

Any suggestions?

OUtput part of fluentd configuration:

  <match **>
    @type copy
    deep_copy true
    <store>
      @type splunk_hec
      protocol https
      hec_host "#{ENV['SPLUNK_HOST']}"
      hec_port "#{ENV['SPLUNK_PORT']}"
      hec_token "#{ENV['SPLUNK_TOKEN']}"
      host "#{ENV['NODE_NAME']}"
      source_key source
      sourcetype_key sourcetype
      <fields>
        pod
        namespace
        container_name
        container_id
        cluster_env
        cluster_name
      </fields>
      <buffer>
        @type memory
        chunk_limit_records 100000
        chunk_limit_size 200m
        flush_interval 5s
        flush_thread_count 1
        overflow_action block
        retry_max_times 3
        total_limit_size 600m
      </buffer>
      <format>
        @type single_value
        message_key log
        add_newline false
      </format>
    </store>
    <store>
      @type prometheus
      <metric>
        (...)
      </metric>
    </store>
  </match>
Labels (1)
0 Karma

RDumbeck
Explorer

Try using namespace::mynamespace

I cannot remember why though.

0 Karma

RDumbeck
Explorer

any luck with this. I have the same problem.

0 Karma

efloss
Engager

Ran into this recently, putting them together in a fields.conf file on the search head will make them searchable without needing the wildcards since they're metadata fields.

[k8s.cluster.name]
INDEXED=true

[k8s.container.name]
INDEXED=true

[k8s.namespace.name]
INDEXED=true

[k8s.node.name]
INDEXED=true

[k8s.pod.name]
INDEXED=true

[k8s.pod.uid]
INDEXED=true

0 Karma
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...