I'm using Spunk Cloud Search & Reporting with Kubernetes 1.25 using Splunk OTel Collector 0.103.0. I have kubernetes pods with multiple containers. Most of the containers have their logs scraped and sent to the splunk index based on the 'splunk.com/index' namespace annotation; so normal Splunk OTEL Collector log scraping. But one of the container's logs must go to a different index. The pods that have a container whose logs must be routed differently have a pod annotation like 'splunk-index-{container_name}=index'. I had this working in Splunk Connector for Kubernetes using this config; ```yaml
customFilters:
#
# filter that set's the splunk_index name from a container annotation
# - The annotation is a concatenation of 'splunk-index-' and
# the name of the container. If that annotation exists on
# the container, then it is used as the splunk index name,
# otherwise the default index is used.
# - This is used in june-analytics and june-hippo-celery to route
# some logs via an annotated sidecar that tails a log file from
# the primary application container.
# - This could be used by any container to specify it's splunk index.
#
SplunkIndexOverride:
tag: tail.containers.**
type: record_transformer
body: |-
enable_ruby
<record>
splunk_index ${record.dig("kubernetes", "annotations", "splunk-index-" + record["container_name"]) || record["splunk_index"]}
</record>
``` My attempt to do this with Splunk OTEL collector uses following config in the values.yaml file for the Splunk OTEL collector v.103.0. Helm chart to add a processor to check for the annotation: ```yaml
agent:
config:
processors:
# set the splunk index for the logs of a container whose pod is annotated with `splunk-index-{container_name}=index`
transform/logs/analytics:
error_mode: ignore
log_statements:
- context: log
statements:
- set(resource.attributes["com.splunk.index"], resource.attributes[Concat("splunk-index-", resource.attributes["container_name"], "")]) where resource.attributes[Concat("splunk-index-", resource.attributes["container_name"], "")] != nil
``` The splunk-otel-collector logs show this error: Error: invalid configuration: processors::transform/logs/analytics: unable to parse OTTL statement "set(resource.attributes[\"com.splunk.index\"], resource.attributes[Concat(\"splunk-index-\", resource.attributes[\"container_name\"], \"\")]) where resource.attributes[Concat(\"splunk-index-\", resource.attributes[\"container_name\"], \"\")] != nil": statement has invalid syntax: 1:65: unexpected token "[" (expected ")" Key*) It seems it does not like the use of Concat() to create a lookup key for attributes. So how would I do this in Splunk OTEL Collector?
... View more
