Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will write about parsing messages to extract valuable information, and then process it consistently across entries. SOCK (Splunk OpenTelemetry Collector for Kubernetes) can be used to process many different kinds of data, one of the most common ones being logs extracted from log files. We use operators to extract and parse information - operators being the most basic units of log processing. As an example, by default, the filelog receiver in the SOCK pipeline uses various operators to extract information from the incoming logs and log file path. This information includes, but is not limited to: namespace, pod, uid, and container name from the log file’s path time, log level, log-tag and log message from the actual log body In later stages of the pipeline, this information is used to enrich the attributes of the log. For example: com.splunk.sourcetype field is set from the container name com.splunk.source field is set from the log file’s path So, if the full path of the container’s log file is: /var/log/pods/kube-system_etcd/etcd/0.log, then com.splunk.source value will be set to this value - we understand the path of the file as its source There might be scenarios where you would like to set a different source other than the default one (i.e. log’s file path) or there is a need to extract some extra attributes from the log message. This article explains how to do it. Operators The OpenTelemetry Collector comes with a set of operators. From README:     An operator is the most basic unit of log processing. Each operator fulfills a single responsibility, such as reading lines from a file, or parsing JSON from a field. Operators are then chained together in a pipeline to achieve a desired result. For instance, a user may read lines from a file using the file_input operator. From there, the results of this operation may be sent to a regex_parser operator that creates fields based on a regex pattern. And then finally, these results may be sent to a file_output operator that writes each line to a file on disk.       Under the hood, SOCK uses a pipeline of several operators to extract the information from the log. We will look at an example of logs produced by containerd - it is one of the runtimes commonly used to run containers (a different runtime could be one like docker). Let’s look at a snippet of an operator from SOCK used to extract data from containerd runtime logs:     - type: regex_parser id: parser-containerd regex: '^(?P<time>[^ ]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$' timestamp: parse_from: attributes.time layout_type: gotime layout: '2006-01-02T15:04:05.999999999Z07:00'   The actual log being read from the file and going into the operator might look like this:   2023-12-27T12:14:05.227298808+00:00 stderr F Hello World   The above operator does a simple thing. It extracts the following data from log messages based on the regular expression:  time: it is set to “2023-12-27T12:14:05.227298808+00:00” stream: “stderr” matches with one of the two possible stream types (stdout or stderr) logtag: is set to “F”  log: “Hello World” - this is an actual log message  Our regex operator extracts these values and inserts them into the event body.   Structure of the log message in the operator pipeline Before we continue, we should learn about the log message format inside the pipeline. Knowing this will help us to apply our own custom operators later. Suppose, we have a slightly different message from containerd:   2023-12-27T12:14:05.227298808+00:00 stderr F Hello World source=xyz   The entry for the above log will look like this in the operator pipeline:   { "timestamp": "2024-05-27T12:21:03.769505512Z", "body": "2024-05-27T12:14:05.227298808+00:00 stderr F Hello World source=xyz", "attributes": { "log": "Hello World source=xyz", "log.iostream": "stderr", "logtag": "F", "time": "2024-05-27T12:14:05.227298808+00:00", }, "resource": { "com.splunk.source": "/var/log/pods/(path to my file)", "com.splunk.sourcetype": "kube:container:(container name)", "k8s.container.name": "(container name)", "k8s.container.restart_count": "0", "k8s.namespace.name": "default", "k8s.pod.name": "(pod name)", "k8s.pod.uid": "(pod uid)", }, "severity": 0, "scope_name": "" }   See how for every info extracted from the log message, there is a corresponding match in the above entry in the attributes field. Regex_parser inserts values into the attributes field by default but this behavior can be changed with the parse_to option. As we can also see there is a log.iostream key in our message, even though we expected stream instead. This is because there is another operator later on in the pipeline that changes it, it looks like this:   - from: attributes.stream to: attributes["log.iostream"] type: move   This operator is used for simple move operations, as we can see it moves the stream field into log.iostream. How do you use custom operators? As an example, let’s consider the same log we saw earlier i.e.   2023-12-27T12:14:05.227298808+00:00 stderr F Hello World source=xyz   What if we want to extract the source from the above message and set it into com.splunk.source resource? Doing that would allow us to assign custom source values based on a log message instead of the path to the file - which is a default behavior. For such a use case, we may create the following operators:   - type: regex_parser id: my-custom-parser regex: '^.*source=(?P<source>[^ ]*).*$' - type: copy from: attributes["source"] to: resource["com.splunk.source"]   If we then use them, the entry for our message will look like this:   { "timestamp": "2024-05-27T12:21:03.769505512Z", "body": "2024-05-27T12:14:05.227298808+00:00 stderr F Hello World source=xyz", "attributes": { "log": "Hello World source=xyz", "log.iostream": "stderr", "logtag": "F", "time": "2024-05-27T12:14:05.227298808+00:00" "source": "xyz", }, "resource": { "com.splunk.source": "xyz", "com.splunk.sourcetype": "kube:container:(container name)", "k8s.container.name": "(container name)", "k8s.container.restart_count": "0", "k8s.namespace.name": "default", "k8s.pod.name": "(pod name)", "k8s.pod.uid": "(pod uid)", }, "severity": 0, "scope_name": "" }   Notice the attribute source, which is parsed by the regex_parser that we just created. This value is then copied into a resource[“com.splunk.source”] by the copy operator.  Using custom operators with values.yaml So, we learned how to create custom operators. But where do we specify them in my_values.yaml to actually use them? Enter extraOperators! For the example discussed above, we will now update our configuration file with the following settings:   logsCollection: containers: extraOperators: - type: regex_parser id: my-custom-parser regex: '^.*source=(?P<source>[^ ]*).*$' - type: copy from: attributes["source"] to: resource["com.splunk.source"]   Now restart the helm deployment and you’re good to go!   helm upgrade --install my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector   Some operators that you might find useful add - can be used to insert either a static value or an expression remove - removes a field, useful for cleaning up unnecessary data after other operations  move - moves (or renames) a field  json-parser - can be useful when you want to parse data saved in a JSON format recombine - combines multi-line logs into one, a topic that we covered extensively in previous blog posts And a lot more can be found here! And some troubleshooting tips So what if I’m not sure what my log entry looks like, I can’t possibly experiment with operators without that knowledge right? Correct! Before experimenting with operators, you should know the structure of your log entry, or else you might end up with faulty data or lots of annoying guesswork. And how would I know the structure of my log entry? You can use stdout operator:   logsCollection: containers: extraOperators: - type: stdout id: my-custom-stdout   Use the above config and restart the helm deployment. Now do kubectl get logs pod_name command and you’ll notice a bunch of logs containing JSON entries. That’s how your entry looks, and how you can debug your operators. Conclusion In this article, we’ve explored some ways of using operators to extract information from the logs. This very powerful feature can be used to parse logs in a more complex way not provided by a basic configuration.  On the other hand, it is important not to overcomplicate things - if you can extract data using built-in functions then do so. SOCK provides many ways to extract data for many commonly used data formats, and using them is much simpler. ... View more

Introduction This blog post is part of an ongoing series on SOCK enablement. In this blog post, I will explain the behavior of SOCK (Splunk Otel Collector for Kubernetes) with the default configuration of values.yaml file. It is a file that contains configurations and variables that get passed to chart templates and dictate how the collector works. A basic way of working with SOCK is to create a new my_values.yaml file and overwrite some configuration values inside of it before passing it to a chart installer. Today we will discuss the default behavior with minimal configuration. How to install it? Structure of values.yaml file Values.yaml file consists of nested variables configuring various system parts. For example, this is a part of the default configuration for the Splunk platform - the core setting responsible for your connection to Splunk: splunkPlatform:  # Required for Splunk Enterprise/Cloud. URL to a Splunk instance to send data  # to. e.g. "http://X.X.X.X:8088/services/collector/event". Setting this parameter  # enables Splunk Platform as a destination. Use the /services/collector/event  # endpoint for proper extraction of fields.  endpoint: ""  # Required for Splunk Enterprise/Cloud (if `endpoint` is specified). Splunk  # Alternatively the token can be provided as a secret.  # Refer to https://github.com/signalfx/splunk-otel-collector-chart/blob/main/docs/advanced-configuration.md#provide-tokens-as-a-secret  # HTTP Event Collector token.  token: ""  # Name of the Splunk event type index targeted. Required when ingesting logs to Splunk Platform.  index: "main"  # Name of the Splunk metric type index targeted. Required when ingesting metrics to Splunk Platform.  metricsIndex: ""  # Name of the Splunk event type index targeted. Required when ingesting traces to Splunk Platform.  tracesIndex: "" (...)   As you can see, various values are used to configure Splunk. These default values are used to set up an application without many extra features, just a Kubernetes logs collection. Other features have to be turned on and configured manually. Comments above each entry are used to describe what it does. As an example, here is a piece of configuration responsible for request timeout: # HTTP timeout when sending data. Defaults to 10s.  timeout: 10s   As we can see, the timeout for sending an event to Splunk is set to 10 seconds. You can customize this value for your own system if there is a need for a longer timeout with your configuration. You can take a look at this documentation to get a better idea about the advanced configuration details, but in this post, we will explore a basic workable configuration and what it does. What kind of minimal configuration do you need to make it work? First, you need to create a my_values.yaml file to overwrite some values in the default configuration. The official documentation explains how to do it: Splunk OpenTelemetry Collector docs. You don’t need many things to run it and a basic configuration file will look something like this: clusterName: "test_cluster" splunkPlatform:  endpoint: "https://X.X.X.X:8088/services/collector/event"  token: "00000000-0000-0000-0000-000000000000"  index: "my_index"  insecureSkipVerify: true   You have to overwrite the clusterName value, as it is required to run the application. It will act as an identifier of your k8s cluster and will be attached to every log, metric, and trace sent, as a k8s.cluster.name attribute. We also have to set the Splunk platform endpoint that will ingest our data and a HEC token that will be used to access it. Refer to this doc on how to set up your HTTP event collector in Splunk. You don’t have to overwrite the index value - by default logs will be sent to the main index -  but it is considered a good practice to do so. In this example, I have changed this value to my_index, an index I created in my instance of Splunk. The logs gathered by SOCK will go to this index. We are setting the insecureSkipVerify flag to true because we want to skip checking a certificate of our HEC endpoint when sending data over HTTPS in our test environment. If you have a certificate configured you should ignore this flag. A short installation guide Great, now we have a working configuration file and can test our application! To install it you first need to add a Splunk otel collector chart with this command: helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart   You should get the response “splunk-otel-collector-chart has been added to your repositories”  if everything was correctly installed. Now that our repo has been added we can install it: helm install my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector As we can see we are installing the my-splunk-otel-collector repo that we just added, with configuration --values from the my_values.yaml file that we’ve created. After running this command we should see a response: Splunk OpenTelemetry Collector is installed and configured to send data to the Splunk Platform endpoint (...) That means our chart was installed correctly! Now that it is running it should be sending data to our Splunk instance using our custom configuration. Another useful command that we can use is this: helm upgrade --install my-splunk-otel-collector --values my_values.yaml splunk-otel-collector-chart/splunk-otel-collector It’s very similar to the command we previously used but can be used even when the repo is already installed and then it will upgrade it with any changes that we made in the my_values.yaml file. What features will be enabled by default? By default, only logs will be sent to Splunk, as metrics and traces have to be turned on manually: logsEnabled: true metricsEnabled: false tracesEnabled: false   So if you want to collect metrics or traces, the values (metricsEnabled and tracesEnabled) need to be changed. If you enabled metrics you will also have to specify metricsIndex, or the application won’t run. If you configured your values correctly and installed your chart, you can now run kubectl get pods command in a console to check if it works. You should see one pod running, something like this: splunker@test:~/splunk-otel-collector-chart$ kubectl get pods  NAME                                   READY   STATUS    RESTARTS   AGE my-splunk-otel-collector-agent-lxns8   1/1     Running   0          63m   As we can see there is one agent pod running because there is only one node in our cluster. In a real-world scenario, you would probably see more agents as there would be more nodes - one agent running per node. And in case you enabled metrics you should be able to also see a cluster receiver pod, like this one: splunker@test:~/splunk-otel-collector-chart$ kubectl get pods  NAME                                                             READY   STATUS    RESTARTS   AGE my-splunk-otel-collector-k8s-cluster-receiver-5d754c9fff-4wzrf   1/1     Running   0          104s my-splunk-otel-collector-agent-fmc9p                             1/1     Running   0          104s   If everything is working fine the status field should state “Running”. So by default our application will send logs (and metrics, traces if enabled) to Splunk. It will also try to resend dropped events in case of failure and use batches to optimize the process of sending data. There are some other commonly used settings like: cloudProvider (aws, azure…) and distribution (aks, eks, openshift…) - these can be used to scrape additional data sendingQueue.persistentQueue - by default, without any configuration, data is queued in memory only. You can enable a persistent queue so queued data is saved on a disk and sent even after collector restarts autodetect - can enable autodetection of prometheus and istio metrics extraAttributes - configuration for additional metadata that can be collected from labels and attributes  So where will my data go? By default logs will go to the “main” index but as mentioned before you can change the index value to send them to a different index. In this example we changed this value to my_index so if you have this index configured in Splunk that’s where data will end up. The simplest way to see if the data is correctly processed is to filter data by index inside Splunk:   There, you should see events being sent to Splunk in real-time. If you want to check your metrics you can use mpreview command like this:   We can observe that the system metrics are now being stored in Splunk. I want to learn more - what other features are there? Many other powerful features of SOCK won't be covered in this article, but there are resources that you can use to learn more about them. You can browse the chart repository and the examples directory to look for ideas for what you can use it for. Reading through values.yaml will also give you a good idea of what can be done with it - all of the settings are described there. Lastly, this series of blog posts is designed to help you learn about SOCK, I recommend taking a look at our other articles, talking about the subjects of routing and multiline logs! 🙂 ... View more

This article is the continuation of the “Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies” blog, where we went through multiline processing for the default Kubernetes logs pipeline.  Let's take a closer look at how the multilineConfigs option functions. This way, you can easily customize it using the standard OTel configuration to fit your specific needs. To fully understand it, we'll go through the operators and break down the default SOCK's filelog configuration into its basic parts. Operators overview The filelog receiver is the critical part of the Splunk OTel collector for Kubernetes log collection mechanism. The receiver is already heavily configured in the helm chart - if you’re interested in how this section looks for your version, run:     kubectl describe cm/<helm-app-name>-otel-agent     And look for the filelog section. The config itself is very long, so today we’ll focus only on the operators section. Thanks to this capability we can create mini pipelines within the receiver itself to be able to process logs correctly based on certain criteria. It might seem like a bunch of complicated, not-understandable statements at first. Don’t worry, we’ll break it down one by one, starting with… Routers! Look at the snippet of the first of SOCK’s filelog receiver operators: Depending on the type of logs, we want to parse it accordingly. In the Kubernetes world, their format depends on the container runtime. In SOCK we support docker, cri-o, and containerd. The routes section is a simple substitute for the switch statement, well-known in the programming world. In the above example, if the log body matches the regex format ^\\{, the log is being passed to another operator with the id of parser-docker, for ^[^ Z]+ it is parser-crio, and ^[^ ]+ goes to parser-containerd. Let’s see one of them, parser-containerd: We can see it is a regex_parser that parses logs to gather fields like time and logtag. And that’s all, then filelog receiver executes operators one by one according to their sequence. But, if you’d like to pass the log to another operator based on your choice, you’d need only to specify the output field. The config would look like this: Parsers As we’ve shown in the previous example, regex_parser parses the string-type field configured in parse_from with regex created by the user. Thanks to that we can extract multiple attributes from one string in one operation. It’s one of many parsers available for filelog operators, another example would be json_parser which is used in SOCK to parse docker logs timestamp: In every parser, there are two options parse_from and parse_to. The default value of parse_from is body - which means a simple logline, and for parse_to it is an array of attributes. In case your logs follow some other popular format, check out other parsing operators like syslog parser or CSV parser. All of them are described here. Recombine Finally, we reached the main point in our operator journey - the recombine operator is very powerful whenever you want to combine consecutive logs into single logs based on simple expression rules. Let’s take one of the examples from our SOCK’s config: This means we combine logs whenever we encounter attributes.logtag set to F. logtag attribute was extracted from the log body before, in parser-containerd, you can go back to this example from the above. Alternatively to is_last_entry you can configure is_first_entry - it should depend on if it’s easier to define the beginning or the end of the multiline block. source_dentifier tells what field should be used to separate one source of logs for combining purposes. In this example, we do it based on the log’s file path. Multiline config for advanced users multilineConfigs setting is fairly easy to use and doesn’t require knowledge about operators, but the drawback is that you can use it only on Kubernetes logs from the default logs pipeline. If you want to set up multiline processing in the pipeline written by you using the extraFileLogs option, you need to configure operators by yourself. Let’s take a look at how the ultimate filelog config looks after applying the multilineConfigs rule, as we’ll need to manually do something similar. We’ll use a Java example here: This values.yaml config produces the following operators snippet: Which can be presented as a diagram:   In both operators, you can see the clean-up-log-record operation which moves attributes.log to body. This is necessary in the case of SOCK because of the processing it does at the beginning with parser-containerd, parser-docker, or parser-crio. The config will be even less complicated if you don’t use such a mechanism. extraFileLogs configuration Let’s start with the setup of a bare logs pipeline: NOTE: preserve_leading_whitespaces option is necessary whenever your processing rule is based on leading whitespaces, if not set OTel will automatically trim your whitespaces. Again we use the same Java log file that produces such a result in Splunk:   This time we need to apply multiline config as a part of filelog operators. Scenario 1 - only recombine In case we’re 100% sure our multiline config applies to all the logs collected by the pipeline, we can use one recombine operator without any complicated logic. For a directory composed of Java files, we can apply such a config: source_identifier is attributes["log.file.path"] as this is the only differentiator we have at this point. Applying this config results in correct log processing:   Scenario 2 - recombine and a router This time let’s combine two cases - java logs and the logs with a timestamp. Such a config requires defining a router to not unnecessarily run all the logs through the same operators. Additionally, in some cases running regex on a log that doesn’t match might result in runtime errors resulting in sending nothing to Splunk. After applying this config:   Both files were processed correctly:     Here an important thing to remember is that operators are being executed one by one, so we have to define noop (the operator that does not do anything) as an exit condition. The diagram looks like this:   General notes There are a few things to remember when creating your filelog pipeline: Every regex pattern must be double-backspaced. When the multilineConfigs option is used, this is automatically done by the Helm mechanism. Remember to do it manually or you might end up with such issues: failed to compile is_first_entry: invalid char escape (1:32) It is important to set output and default fields in routers. For example - if we didn’t do it in the previous example and the log would match the first expression it would be passed to the newline-processor, but then the timestamp-processor would be triggered as well - because it is next in a sequence. There are better ways to create expression patterns than attributes["log.file.path"] matches ".*java.*", we should generally avoid greedy regexes whenever possible. You can learn more about expression language to find out a better way that suits your needs.   ... View more

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The default behavior of filelog receiver is to send log lines one by one. It is the desired behavior in most cases, but it’s problematic whenever the information is too long to be kept in one line only. This can lead to fragmented and incomplete log entries, making it challenging to analyze or interpret the logged data coherently. In this blog, you’ll learn how to configure multilineConfigs in SOCK (Splunk Otel Collector for Kubernetes) to handle multiline log entries appropriately, ensuring that complex information is presented in a structured and readable manner. Problem statement Let’s see the example of a Java exception sent to Splunk by SOCK: If we click Event Actions > Show Source (with Wrap results enabled) we’ll see such a log snippet:   So we can see that the one-by-one approach doesn’t work in this case. The solution - java traces Even though the recombine operator is the right tool to use in this case, we don’t need to prepare the config on our own as this is already simplified by the multilineConfigs option in SOCK. Check out the example below, and read more about it in this documentation.  My java pod comes from the deployment java-app-deployment (in default namespace) and its container static name is java-app-container, so the config above is enough to identify all the logs that must be processed and recombined according to the pattern. Alternatively, we could configure something like this: Here we use podName as the main datapoint that identifies the log source, but it is defined as regex (because the pod names contain dynamic IDs that are different every time the pod is triggered). Because using regex is resource intensive, we should avoid using it whenever we can, so the first version of the config is much better than the second one. The most important thing here is the firstEntryRegex that specifies what pattern regex should find a match to treat the line as the first line of the block sequence. In this case, the example is ^[^\s] which means “it shouldn’t start with the whitespace character”. Let’s see it on the regex101:   Note that you don’t need to create a regex to match the whole line - the beginning of it is enough. Now we’re ready to apply the configuration and see the result in Splunk:   Finally, it looks exactly how it should be. The solution - curl outputs Another example might be the processing of outputs of a system command - curl, a powerful tool for making HTTP requests and retrieving data from URLs. This is how it looks in Splunk:   The source is:   We can see that curl blocks start with either <HTML> or the <html> marker. The multiline config in this case might look like this: As you can see the firstEntryRegex is defined as (^<html>)|(^<HTML). After applying the config the result in Splunk has changed to be:   The solution - logs with the timestamp Logs likely include a timestamp at the beginning of every line. This is a good sign for us whenever we want to process it with a multiline config, as timestamps are easy to express as regexes. The example below shows such a log of a Java application sent without any configured processing to Splunk:   This is a similar case to the one from above, but this time, simple ^[^\s] won’t work. Looking for the whitespaces as the beginning groups exceptions together, but still, it doesn’t connect it with the proper beginning of the log - the timestamp. Using regex101 we can create the following regex:   And incorporate it into the SOCK configuration: After reloading, the logs are correctly grouped: Conclusion   This article explored multiple ways of combining multiline log entries into one. This is a useful feature, as such logs are commonly encountered, and parsing them correctly greatly helps with working with them down the line. ... View more

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest metrics, traces, and logs to the Splunk platform using an HEC. If you are a DevOps Engineer or SRE, you may already be familiar with the OTel Collector’s flexibility, but for those less experienced, this blog post will serve as an introduction to routing logs.  The idea of OpenTelemetry as a whole is to unify the data so it's suitable for every input and output and put some processors in between to make it possible to perform operations on data (such as transforming and filtering). You may already see that one of the biggest advantages of OTel Collector is its flexibility - but sometimes figuring out how to use it in practice is a challenge. One of the most common cases in log processing is setting up the event’s index. If you’re familiar with the Splunk HEC receiver, you might recall this configuration snippet: This indicates that every event used by this exporter will be sent to the logs index.  As you may see, the logs index is specific to an exporter, so the intuition is to create as many splunk_hec exporters as you need, and additionally create multiple filelog receivers as well, so that we can filter which files go to which index. Using your imagination, visualize a scenario where all the logs go to the ordinary logs index, but some are only visible to people with higher permissions levels. These logs are gathered by filelog/security receiver and the pipeline structure would look like this one: But is it really the best solution? Let’s consider a few questions here: splunk_hec exporter config seems to be the same, the only difference is the index field. Does it make sense to copy the configuration over and over? filelog receiver gives a way of configuring a place to gather logs. What about other filtering options, like ones based on severity or a specific phrase in the log’s body? Everytime we create a new pipeline, a new process comes to life - doesn’t this consume too many resources? The solution: Dynamic index routing Today I’ll show you how to create a pipeline with dynamic index routing, meaning it is based on incoming logs and not statically set, with a transform processor and Splunk OpenTelemetry Collector for Kubernetes (SOCK). The idea is based on this attribute from Splunk HEC Exporter documentation: This means that we can specify com.splunk.index as a resource attribute for a log, and it will overwrite the default index. Let’s go through a few examples of how we can do it in SOCK. Viewing the pipelines configured in SOCK Before we cover how to overwrite your pipelines, let’s start with how you can view the pipeline. The final config is the result of your configuration in values.yaml, as well as the default configuration that is delivered by SOCK. The config’s yaml file is in the pod’s configmap. As logs are generated by the agent, you can look at the agent’s config, the command is: Where my-splunk-otel-collector-otel-agent is the configmap’s name - it might differ in your case, especially if you chose a different name for an installation versus one from the Getting Started docs. You can take a look at a configmaps you have with the command: An output example for a default namespace would be: After successfully running the describe command, scroll all the way down until you see the pipelines section. For logs, it looks more or less like this: Now you know what components your logs pipeline is made of!  Easy scenarios Now let’s get our hands dirty! Let’s see the easy examples of index routing based on real scenarios. Scenarios based on the log attribute The scenario: Let’s say we want to pass all the events with a log.iostream attribute stderr to error_index This would capture events emitted to the error stream and send them to their own index. The solution: This requires doing two things: Overwriting the default agent config with a new transform processor and adding it to the logs pipeline.  Setting up the new processor’s statement to specify the correct target index. Every transform processor consists of a set of statements. We need to create one that matches our use case, by defining what we need and writing it specifically for OTel. The logical statement here would be: set com.splunk.index value to be error_index for EVERY log from the pipeline whose attribute log.iostream is set to stderr Then the statement in the transform processor’s syntax described here looks like this: Next, we need to append the processor to the logs pipeline. To do that, we need to copy and paste the current processors under the agent.config section then insert our processor at the end. The whole config will be: After applying the config, the stderr events appear in the error_index: Scenarios based on specific log text The scenario Passing an event to a different index when something specific is written in the body of the log, for example, every log that contains [WARNING]: The solution All the keywords used here come from the transform processor documentation. We can use the  transform processor, this time using the following logic: Here are some sources that can be used to learn more about OpenTelemetry Transformation Language and its grammar. Then we repeat the steps described in the previous solutions section. The final config is: And the result in the Splunk Enterprise looks like this: How do I know what attributes I can use? At this point, you might think “Oh right, that looks easy, but how would I know what attributes to use?” The logs in the transform processor can use all the elements described here, but the most useful ones are: body - referring to the log body attributes - referring to the attributes specific to a single log resource.attributes - referring to the attributes specific to multiple logs from the same source You can see them in the Splunk Enterprise event preview: However, there’s no indication as to which dimensions are attributes and which are resource.attributes. You can see how it looks by running  your OTel agent with this config: This will produce all the information about log structure and which attributes are really the resource.attributes: From this snippet, you can see that only logtag and log.iostream are attributes, all the rest are part of the resource.attributes. The transform processor has many options aside from the ones described above, check them out here.  Complex Scenarios Let’s go even deeper and operate on two variables instead of one. Scenarios based on setting index-based annotations You may want to annotate the whole namespace with one splunk.com/index, but want specific pods from this namespace to redirect somewhere else. You can do this by using a transform processor to provide additional annotations to the pod of your choice. Let’s say the annotation is second_index. This is how it looks in kubectl describe of the pod: Transforming annotation into resource.attribute First, redirect logs from the pods according to the  second_index annotation to convert the annotation to a resource.attribute. This can be done with extraAttributes.fromAnnotations config:   tag_name is the identifier of an element in resource.attributes, it is optional. If you don’t configure it your attribute will look like this: k8s.pod.annotations.<key>is the output format. With tag_name you can decide how the name of your attribute, in this example it is the same as the key: Make OTel pass logs to the index Now that we have resource.attribute second_index set up, we can set the index destination for logs. We will use transform processor for this purpose: We will replace the com.splunk.index resource attribute with the second_index attribute, but only when the second_index attribute is present - so it doesn’t affect logs from other pods. Delete unnecessary attributes Once the attribute has been moved to the log's index, we can get rid of it. This requires adding another statement to the transform processor: Scenarios based on labels setting the index  This will work exactly the same as an annotation example from the previous section, the only difference is in how we’re transforming the label into resource.attribute. We now have the  second_index label on a pod: We can make it visible to the OTel collector with this config snippet: Conclusion In this article, I showed you how to route logs to different indexes. It is a commonly used feature and it can be used in many scenarios, as we can see in the examples. We will expand on other SOCK features in later articles, so stay tuned! ... View more