Hi,
We use splunk-connect-for-kubernetes to send logs to splunk via HEC mechanism. Sending logs to splunk is fine, but searching is not.
When we search for
namespace=mynamespace "*Exception*"
There is lots of missing logs, very few is returned.
But, when I search like that:
namespace=*mynamespace* "*Exception*"
All is fine, all logs are returned
Any suggestions?
OUtput part of fluentd configuration:
<match **>
@type copy
deep_copy true
<store>
@type splunk_hec
protocol https
hec_host "#{ENV['SPLUNK_HOST']}"
hec_port "#{ENV['SPLUNK_PORT']}"
hec_token "#{ENV['SPLUNK_TOKEN']}"
host "#{ENV['NODE_NAME']}"
source_key source
sourcetype_key sourcetype
<fields>
pod
namespace
container_name
container_id
cluster_env
cluster_name
</fields>
<buffer>
@type memory
chunk_limit_records 100000
chunk_limit_size 200m
flush_interval 5s
flush_thread_count 1
overflow_action block
retry_max_times 3
total_limit_size 600m
</buffer>
<format>
@type single_value
message_key log
add_newline false
</format>
</store>
<store>
@type prometheus
<metric>
(...)
</metric>
</store>
</match>
Try using namespace::mynamespace
I cannot remember why though.
any luck with this. I have the same problem.
Ran into this recently, putting them together in a fields.conf file on the search head will make them searchable without needing the wildcards since they're metadata fields.
[k8s.cluster.name]
INDEXED=true
[k8s.container.name]
INDEXED=true
[k8s.namespace.name]
INDEXED=true
[k8s.node.name]
INDEXED=true
[k8s.pod.name]
INDEXED=true
[k8s.pod.uid]
INDEXED=true