How to modify logs via Splunk Otel Collector?

[ppal]

Hi Everyone,

I am trying to replicate log modification that was possible with fluentd when using splunk-connect-for-kubernetes.

splunk_kubernetes_logging: 
   cleanAuthtoken:
        tag: 'tail.containers.**'
        type: 'record_modifier'
        body: |
            # replace key log
            <replace>
              key log
              expression /"traffic_http_auth".*?:.*?".+?"/
              # replace string
              replace "\"traffic_http_auth\": \"auth cleared\""
            </replace>

Now since the above charts support ended we have switched to splunk-otel-collector.
Along with this we also switched the logsengine: otel  and now having a hard time replicating this modification.

Per the documentation I read this should come via processors (which is the agent), please correct me if I am wrong here. I have tried two processors but both doesn't work.  What I am missing here?

logsengine: otel
agent:
 enabled: true
 config:
   processors:
      attributes/log_body_regexp:
        actions:
          - key: traffic_http_auth
            action: update
            value: "obfuscated"
      transform:  
        log_statements:
          - context: log
            statements:
              - set(traffic_http_auth, "REDACTED")

This is new to me, can anyone point me where this logs modifiers can be applied. 

Thanks,
Ppal

[PaulPanther]

@ppal Have you added these both processors under service --> pipelines --> <your log-pipeline> --> processors as well? 

[ppal]

Yes, it does. 

    service:
      pipelines:
        logs:
          exporters:
          - otlp
          processors:
          - transform
          - attributes/upsert

So far I have tried these options but none seem to work. 

processors:
  attributes/upsert:
    actions:
      - key: upstream_namespace
        action: upsert
        value: "REDACTED_NS"
  transform:  
    log_statements:
      - context: log
        statements:
          - replace_all_patterns(attributes,"value","upstream_namespace", "REDACTED_NS")
          - replace_all_patterns(attributes,"key","upstream_namespace", "REDACTED_NS")
          - replace_match(attributes["upstream_namespace"], "*" , "REDACTED_NS")
          - replace_match(attributes["upstream_namespace"], "system-monitoring" , "REDACTED_NS")
          - delete_key(attributes,"upstream_namespace")
          - delete_key(resource.attributes,"upstream_namespace")
          - replace_all_patterns(attributes["upstream_namespace"],"value","upstream_namespace", "REDACTED_NS")
          - replace_all_patterns(attributes["upstream_namespace"],"value","system-monitoring", "REDACTED_NS")

The attribute/upsert and set() however appends to existing value. 

upstream_namespace: REDACTED_NS
                    system-monitoring

Not sure what is missing here, any suggestions to resolve this?

Thanks

[PaulPanther]

Could you please activate only the attributes in your pipeline "logs", get rid of the transforms block and then verify the functionality?

Next time it would be great if we can focus on one configuration that does not work.

[ppal]

yes, each method was tested separately to it doesn't overlap.

I just combined it here to it's easier to see what been tried so far.

Here is the sample log (reduced to a few kv) from k8s nodes that is pulled by collector(agent). 

2024-03-11T21:04:41.411025006Z stdout F {"time": "2024-03-11T21:04:41+00:00", "upstream_namespace":"system-monitoring", "remote_user": "sample-user"}

 
If for example I just use `attributes/upsert` it appends to existing but not overwrite it.

[PaulPanther]

If you wanna manipulate the raw log event I'm pretty sure you have to do it with props & transforms or a data stream solution.

As far as I know no otel processor supports the mentioned fluentd functionality. 

[ppal]

Interesting, thanks for taking time and replying to my queries. @PaulPanther