Splunk Dev

Syntax in OTEL for setting log index based on K8s annotation

webered
New Member

I'm using Spunk Cloud Search & Reporting with Kubernetes 1.25 using Splunk OTel Collector 0.103.0.  I have kubernetes pods with multiple containers.  Most of the containers have their logs scraped and sent to the splunk index based on the 'splunk.com/index' namespace annotation; so normal Splunk OTEL Collector log scraping.  But one of the container's logs must go to a different index.  The pods that have a container whose logs must be routed differently have a pod annotation like 'splunk-index-{container_name}=index'.  I had this working in Splunk Connector for Kubernetes using this config;

 

 

```yaml
  customFilters:
    #
    # filter that set's the splunk_index name from a container annotation
    # - The annotation is a concatenation of 'splunk-index-' and
    #   the name of the container.  If that annotation exists on
    #   the container, then it is used as the splunk index name,
    #   otherwise the default index is used.
    # - This is used in june-analytics and june-hippo-celery to route
    #   some logs via an annotated sidecar that tails a log file from
    #   the primary application container.
    # - This could be used by any container to specify it's splunk index.
    #
    SplunkIndexOverride:
      tag: tail.containers.**
      type: record_transformer
      body: |-
        enable_ruby
        <record>
          splunk_index ${record.dig("kubernetes", "annotations", "splunk-index-" + record["container_name"]) || record["splunk_index"]}
        </record>
```

 

 

My attempt to do this with Splunk OTEL collector uses following config in the values.yaml file for the Splunk OTEL collector v.103.0. Helm chart to add a processor to check for the annotation:

 

 

```yaml
agent:
  config:
    processors:
      # set the splunk index for the logs of a container whose pod is annotated with `splunk-index-{container_name}=index`
      transform/logs/analytics:
        error_mode: ignore
        log_statements:
          - context: log
            statements:
              - set(resource.attributes["com.splunk.index"], resource.attributes[Concat("splunk-index-", resource.attributes["container_name"], "")]) where resource.attributes[Concat("splunk-index-", resource.attributes["container_name"], "")] != nil
```

 

 

 The splunk-otel-collector logs show this error:

Error: invalid configuration: processors::transform/logs/analytics: unable to parse OTTL statement "set(resource.attributes[\"com.splunk.index\"], resource.attributes[Concat(\"splunk-index-\", resource.attributes[\"container_name\"], \"\")]) where resource.attributes[Concat(\"splunk-index-\", resource.attributes[\"container_name\"], \"\")] != nil": statement has invalid syntax: 1:65: unexpected token "[" (expected ")" Key*)

It seems it does not like the use of Concat() to create a lookup key for attributes.  So how would I do this in Splunk OTEL Collector?

Labels (1)
0 Karma
Get Updates on the Splunk Community!

Splunk App for Anomaly Detection End of Life Announcement

Q: What is happening to the Splunk App for Anomaly Detection?A: Splunk is officially announcing the ...

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...