Hi, We use splunk-connect-for-kubernetes to send logs to splunk via HEC mechanism. Sending logs to splunk is fine, but searching is not. When we search for
namespace=mynamespace "*Exception*"
There is lots of missing logs, very few is returned. But, when I search like that:
namespace=*mynamespace* "*Exception*"
All is fine, all logs are returned
Any suggestions?
OUtput part of fluentd configuration:
<match **>
@type copy
deep_copy true
<store>
@type splunk_hec
protocol https
hec_host "#{ENV['SPLUNK_HOST']}"
hec_port "#{ENV['SPLUNK_PORT']}"
hec_token "#{ENV['SPLUNK_TOKEN']}"
host "#{ENV['NODE_NAME']}"
source_key source
sourcetype_key sourcetype
<fields>
pod
namespace
container_name
container_id
cluster_env
cluster_name
</fields>
<buffer>
@type memory
chunk_limit_records 100000
chunk_limit_size 200m
flush_interval 5s
flush_thread_count 1
overflow_action block
retry_max_times 3
total_limit_size 600m
</buffer>
<format>
@type single_value
message_key log
add_newline false
</format>
</store>
<store>
@type prometheus
<metric>
(...)
</metric>
</store>
</match>
... View more