This blog post is part of an ongoing series on OpenTelemetry. Heroku is a hosting service offering to run your application in a self-contained environment that you can scale over time. Heroku is impressively simple to use, with the ability to push to deploy by just pushing to their remote git repository. We have been building support for Heroku by adding a buildpack that installs the Splunk distribution of the OpenTelemetry collector to your Heroku application (called a slug). In this blog post, we will show you how to set up your Heroku app to use our buildpack. We will show off a sample app that is able to use the collector to report metrics, traces, and profiling information to the Observability Cloud. Prerequisites: You have installed the Heroku CLI. You have created a Heroku app. All commands we run will run in the checkout of your app as a git repository. Our buildpack is open source and hosted on Github under this repository: https://github.com/signalfx/splunk-otel-collector-heroku Install the buildpack in your Heroku app with:   heroku buildpacks:add https://github.com/signalfx/splunk-otel-collector-heroku.git#v0.0.3   The #0.0.3 segment here represents the current version of the buildpack. Make sure to change it as needed to target the latest version. Next, obtain an access token to send data to the observability cloud and set it, as well as the realm you operate in:   heroku config:set SPLUNK_ACCESS_TOKEN=1234 heroku config:set SPLUNK_REALM=us0   Optionally, you may decide to reduce the disk space used by the collector by removing the Smart Agent bundle.   heroku config:set SFX_AGENT_DISABLED=true   Now, you can deploy the app. We offer a test application under our test folder: https://github.com/signalfx/splunk-otel-collector-heroku/tree/main/test You can copy the contents of this application into your git repository, commit and push them to heroku to try it out:   cp -R <checkout>/test/* . git add -A git commit -m “initial commit” git push heroku main   This will show you the deployment of the application to heroku. You can then go to your application HTTP URL and you will be greeted with: We instrument node with the -r argument loading this file:   const { start } = require('@splunk/otel'); const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http'); const { getInstrumentations } = require('@splunk/otel/lib/instrumentations'); delete process.env.SPLUNK_REALM; start({ endpoint: 'http://localhost:4317', metrics: { exportIntervalMillis: 30000, // default: 5000 }, profiling: { memoryProfilingEnabled: true }, tracing: { instrumentations: [ ...getInstrumentations(), new HttpInstrumentation({ headersToSpanAttributes: { // Server side capturing, e.g. express server: { // Outgoing response headers responseHeaders: ['content-type'], // Incoming request headers requestHeaders: ['accept-language'] }, // Client side capturing, e.g. node-fetch, got client: { // Incoming response headers responseHeaders: ['server-timing'], // Outgoing request headers requestHeaders: ['accept-encoding'] } } }), ], }, });   In this file, we set the endpoint to send all data to our localhost over HTTP with port 4317. That’s where our collector resides. The data will be transmitted over the OTLP protocol JSON format. We enable metrics, profiling, and tracing for our application with a few customizations to get more information from Express, the Node.js framework we use to respond to HTTP requests. First, we use a customer visits metric as a counter that is incremented every time you visit. The metric is annotated with information from your heroku environment such as app id and name, revision number, release tag, and slug commit. We collect a trace of your visit. We use auto-instrumentation to instrument Express and a custom tracer to pick your visit specifically. You can zoom in on a span to get more information about your application metadata. Since profiling is on, we can also visit the CPU profile of our application: We can also visit a beautiful flame chart of the memory allocation of Node for our application: That’s it! If you have any questions, please make sure to ask in the community forum or open an issue with our github repository. Thanks for reading! — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. In this blog post, we will explore the best way to check your connection to the HEC endpoint of your Splunk Cloud or Splunk on-premises deployment. HEC stands for HTTP Event Collector, and is described at length in our documentation. HEC is a staple of Splunk - it was first introduced at .conf 2015, with a fun demo using the audience’s phones. Before we start, you should check where your HEC endpoint is located. If you are on a Splunk Cloud instance, your endpoint may use a different hostname. The documentation gives you a complete rundown. If you are using a Splunk Cloud Platform instance, the URL typically looks like https://http-inputs-<host>.splunkcloud.com/services/collector/event, where <host> is replaced with your Splunk Cloud instance name.  Please make sure to follow the documentation! This is a common source of confusion. To dissipate confusion further, you can check that your HEC endpoint is valid and ready to use with a call to the health endpoint of your HEC server. Use this command to interact with the server (replacing FOO with your Splunk Cloud instance name):   curl "https://http-inputs-FOO.splunkcloud.com/services/collector/health"   If this is the correct endpoint, you will receive a successful response with the following body: {"text":"HEC is healthy","code":17} This validates that you are targeting the correct host. You still may need to proceed further to check that you can indeed send data to Splunk. To do so, you can create a curl command that will send a HEC event to the server. The documentation offers such an example:   curl "https://mysplunkserver.example.com:8088/services/collector" \ -H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \ -d '{"event": "Hello, world!", "sourcetype": "manual"}'   Note the Authorization header. You would replace its value with the access token you would define in your Splunk environment. Please make sure your token is allowed to send data to the correct indexes. You can then check that the data was indeed ingested by searching Splunk with index=* sourcetype=manual. I hope this example was useful! Please feel free to reply in your comments with your own tips and tricks or provide feedback on the documentation. Your suggestions are most welcome! — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. For this blog post, we customize a Splunk OTEL Collector configuration file to add a logging exporter. We are sending metrics to signalfx-forwarder endpoint and instead of sending it to the backend, we are just logging it as debug output.   exporters: # Traces sapm: access_token: "${SPLUNK_ACCESS_TOKEN}" endpoint: "${SPLUNK_TRACE_URL}" # Metrics + Events signalfx: access_token: "${SPLUNK_ACCESS_TOKEN}" api_url: "${SPLUNK_API_URL}" ingest_url: "${SPLUNK_INGEST_URL}" # Use instead when sending to gateway #api_url: http://${SPLUNK_GATEWAY_URL}:6060 #ingest_url: http://${SPLUNK_GATEWAY_URL}:9943 sync_host_metadata: true correlation: # Logs splunk_hec: token: "${SPLUNK_HEC_TOKEN}" endpoint: "${SPLUNK_HEC_URL}" source: "otel" sourcetype: "otel" # Send to gateway otlp: endpoint: "${SPLUNK_GATEWAY_URL}:4317" tls: insecure: true # Debug logging: loglevel: debug service: extensions: [health_check, http_forwarder, zpages, memory_ballast] pipelines: metrics: receivers: [smartagent/signalfx-forwarder, signalfx] processors: [memory_limiter, batch, resourcedetection] # exporters: [signalfx] exporters: [logging] # Use instead when sending to gateway #exporters: [otlp]   Check in particular how we modified the metrics pipeline. Start Splunk OTEL Collector using docker command and start sending output to output.log file, so that you are not overwhelmed by the amount of data on your screen:   docker run --rm -e SPLUNK_ACCESS_TOKEN="INGEST_TOKEN_VALUE" \ -e SPLUNK_HEC_TOKEN="INGEST_TOKEN_VALUE" \ -e SPLUNK_CONFIG=/etc/collector.yaml -p 13133:13133 -p 14250:14250 \ -e SPLUNK_BALLAST_SIZE_MIB=2666 -e SPLUNK_MEMORY_LIMIT_MIB=8000\ -e SPLUNK_API_URL="https://api.us1.signalfx.com" \ -e SPLUNK_TRACE_URL="https://ingest.us1.signalfx.com/v2/trace" \ -e SPLUNK_INGEST_URL="https://ingest.us1.signalfx.com" \ -e SPLUNK_HEC_URL="https://ingest.us1.signalfx.com/v1/log" \ -p 14268:14268 -p 4317:4317 -p 4318:4318 -p 6060:6060 -p 8888:8888 \ -p 9080:9080 -p 9411:9411 -p 9943:9943 \ -v "${PWD}/agent_config.yaml":/etc/collector.yaml:ro \ --name otelcol quay.io/signalfx/splunk-otel-collector:latest \ &> output.log   Depending on the use case, you can omit exposing some of the ports.  Also, you can update URLs as per your realm, if you are sending data to the backend. Open up a new command line tab and run the following curl command, to send test metric datapoint to Splunk OTEL Collector:   curl -X POST "http://localhost:9080/v2/datapoint" \ -H "Content-Type: application/json" \ -H "X-SF-Token: ENTER_TOKEN" \ -H "Connection: Keep-Alive" \ -H "Keep-Alive: timeout=5, max=100" \ -d '{ "gauge": [ { "metric": "test", "value": 1 } ] }'   You should see the same data in output.log file as following:   Resource SchemaURL: https://opentelemetry.io/schemas/1.6.1 Resource attributes: -> host.name: STRING(1099f15c9e2b) -> os.type: STRING(linux) ScopeMetrics #0 ScopeMetrics SchemaURL: InstrumentationScope Metric #0 Descriptor: -> Name: test -> Description: -> Unit: -> DataType: Gauge NumberDataPoints #0 Data point attributes: -> system.type: STRING(signalfx-forwarder) StartTimestamp: 1970-01-01 00:00:00 +0000 UTC Timestamp: 2022-11-03 20:14:24.241657411 +0000 UTC Value: 1 {"kind": "exporter", "data_type": "metrics", "name": "logging"}   Now you can see exactly what is being sent to our backend! You can use this method to send custom metrics to your backend as well. — Maulik Patel, Professional Services Senior Consultant at Splunk ... View more

This blog post is part of an ongoing series on OpenTelemetry. This post shows how to pre-process logs to add additional metadata, such as source type, index or host.name. WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud. The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF) The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, and Helm among others. OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes, and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. Please see our official documentation to install the collector. At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples. In this example, we are going to use our former blog post, ingesting logs from a file and sending them to Splunk Enterprise. We are going to apply a twist to this example by creating three pipelines that read from three different files. Data coming from those three files will be associated with different source types, depending on their source. A source type is a default field that identifies the structure of an event. A source type determines how Splunk Enterprise formats the data during the indexing process. Our Splunk HEC exporter reads a specific element of the log to determine the source type to output in HEC events. Per the specification of the log data model, it uses the com.splunk.sourcetype resource attribute. You can override the attribute to look for in the HEC exporter configuration with hec_metadata_to_otel_attrs/sourcetype. A resource is a group of log records - so we don’t repeat the same metadata for each log entry. To manipulate resource attributes, we can use a processor named the resources processor. This is a typical use of the processor to set the attribute com.splunk.sourcetype to the value sourcetype1.       resource/one: attributes: - key: com.splunk.sourcetype value: "sourcetype1" action: upsert       OK, so our example from there consists of adding this processor, configured differently, to all three pipelines:       processors: batch: resource/one: attributes: - key: com.splunk.sourcetype value: "sourcetype1" action: upsert resource/two: attributes: - key: com.splunk.sourcetype value: "sourcetype2" action: upsert resource/three: attributes: - key: com.splunk.sourcetype value: "sourcetype3" action: upsert pipelines: logs/one: receivers: [ filelog/onefile ] processors: [ batch, resource/one ] exporters: [ splunk_hec/logs ] logs/two: receivers: [ filelog/twofile ] processors: [ batch, resource/two ] exporters: [ splunk_hec/logs ] logs/three: receivers: [ filelog/threefolder ] processors: [ batch, resource/three ] exporters: [ splunk_hec/logs ]       We have put this all together into an example that lives under Splunk’s OpenTelemetry Collector github repository. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed. First, check out the repository using git clone: https://github.com/signalfx/splunk-otel-collector.git Using a terminal window, navigate to the folder examples/otel-logs-with-sourcetypes-splunk. Type: docker-compose up This will start the OpenTelemetry Collector, our bash script generating data, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display the same information as in our last blog post to let us know it is ready. Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme. You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. In the search box, enter index=logs to start searching for your logs. You will see the logs have been ingested with separate source types: When you have finished exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! This concludes our Prometheus metrics example. We have used the OpenTelemetry Collector to successfully scrape Prometheus metrics and send them to our Splunk Enterprise instance. If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository. — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. This blog post shows how you can use the OpenTelemetry Collector, a Discord bot library, and create a cool solution to follow Discord activity or post alerts. Discord is a free SaaS forum software used by cool kids to communicate. And OpenTelemetry is a great way to collect traces, metrics, and logs. To get started, we create a bot following this tutorial which explains how to create a simple Discord bot in Golang. There are two parts to this example. First, you create an application in Discord using the developers portal and a bot associated with it. Second, you write just enough go code to make the bot reply pong when it sees the message "ping". Taking this example as inspiration, we can create our own bot that is able to handle messages for starters.   func (b *botImpl) Start() error { log.Info().Msg("Starting bot") goBot, err := discordgo.New("Bot " + b.token) if err != nil { return err } goBot.AddHandler(b.messageHandler) err = goBot.Open() if err != nil { return err } log.Info().Msg("Started bot") b.goBot = goBot return nil }   This bot defines a message handler function, messageHandler:   func (b *botImpl) messageHandler(s *discordgo.Session, m *discordgo.MessageCreate) { data, err := json.Marshal(m) if err != nil { log.Error().Err(err).Msg("Error marshaling message") return } b.sink(m.Timestamp, data) }   This call to b.sink calls out to a function passed into the bot:   sink func(timestamp time.Time, data []byte)   That's it for our bot for now. The sink function itself calls out to a "hecClient":   func(timestamp time.Time, bytes []byte) { err := hecClient.SendData(timestamp, bytes) if err != nil { logger.Error("Error sending data", zap.Error(err)) } }   The HEC client is a simple wrapper for the Splunk HEC exporter component from OpenTelemetry. We create and start our exporter programmatically like so:   exporter, err := factory.CreateLogsExporter(context.Background(), component.ExporterCreateSettings{ TelemetrySettings: component.TelemetrySettings{ Logger: logger, TracerProvider: trace.NewNoopTracerProvider(), MeterProvider: nonrecording.NewNoopMeterProvider(), MetricsLevel: configtelemetry.LevelNone, }, BuildInfo: component.NewDefaultBuildInfo(), }, hecConfig) if err != nil { return nil, err } if err = exporter.Start(context.Background(), componenttest.NewNopHost()); err != nil { return nil, err }   We wrap the exporter with a batch processor so we'll send multiple events at once if possible:   batchProcessorFactory := batchprocessor.NewFactory() processor, err := batchProcessorFactory.CreateLogsProcessor(context.Background(), componenttest.NewNopProcessorCreateSettings(), batchProcessorFactory.CreateDefaultConfig(), exporter) if err != nil { return nil, err } if err = processor.Start(context.Background(), componenttest.NewNopHost()); err != nil { return nil, err }   Since we're outside the collector, we stub some of the telemetry and configuration settings available. To test out this bot, we create a Discord application and create a bot there with a unique ID that we drop in our configuration. We invite our bot to a channel of our choice and send messages to it - we can see the messages show up in Splunk in real-time as the bot is able to read them. The messages are quite comprehensive as they contain author information and metadata. One more thing - since our bot can listen for messages, it can send them as well. We create a HTTP server exposed by the bot that can process webhook actions from Splunk alerts.   listen := s.listenAddr if listen == "" { listen = httpAddr } s.webServer = &http.Server{ Addr: listen, Handler: s, ReadTimeout: serverReadTimeout, WriteTimeout: serverWriteTimeout, IdleTimeout: serverIdleTimeout, } err := s.webServer.ListenAndServe() if err != http.ErrServerClosed { return err } return nil   The server takes a handler method that processes webhooks and posts to the channel ID associated, if found:   wh := req.URL.Query().Get("webhook") var cfg *config.WebhookConfig for _, whc := range s.webhooks { if whc.ID == wh { cfg = whc break } } if cfg == nil { resp.WriteHeader(404) s.logger.Debug("no webhook defined", zap.String("remoteAddr", req.RemoteAddr), zap.String("webhook", wh)) return } var alertRequest AlertRequest err := json.NewDecoder(req.Body).Decode(&alertRequest) if err != nil { s.logger.Debug("bad request", zap.String("remoteAddr", req.RemoteAddr), zap.Error(err)) resp.WriteHeader(400) return } err = s.bot.SendMessage(cfg.Channel, fmt.Sprintf("%s - see results %s", alertRequest.SearchName, alertRequest.ResultsLink)) if err != nil { s.logger.Error("error sending message to Discord", zap.String("remoteAddr", req.RemoteAddr), zap.Error(err)) resp.WriteHeader(500) return } resp.WriteHeader(200)   The  webhook config comes from our configuration file:   { "token": "REPLACE WITH DISCORD BOT TOKEN", "hec_endpoint": "http://localhost:8808/collector/event", "hec_token": "00000000-0000-0000-0000-0000000000000", "hec_index": "main", "hec_insecure_skip_verify": true, "listen_addr": "0.0.0.0:8080" "webhooks": [ { "id": "foo", "channel": "REPLACE WITH CHANNEL ID" } ] }   Note the "foo" id. Now in Splunk, we define an alert that will call the webhook foo. We create a real-time alert reacting to a search for the term "boo". The alert will call out to a webhook with http://bot:8080/?webhook=foo. To put this together, we can try to type "boo" in Discord - this message gets ingested in Splunk, which reacts by posting an alert to our Discord channel: That’s all folks! The code is available under Apache 2.0 License for your sheer enjoyment. Your patches are very welcome. — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF). The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, or Helm among others. OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes, and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples. This blog post will walk you through using the Splunk HEC raw endpoint to send data to the OpenTelemetry collector. WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud. When Splunk announced the HTTP Event Collector, it created a way for folks to send structured data using JSON payload describing an event taking place in their infrastructure. Typically, events contain the timestamp, source, sourcetype, and of course the payload of the event. Optionally, customers can define the index of the event so it is routed properly. Sometimes though, you don’t have the luxury to encapsulate data. Splunk HEC supports sending data to a specific path (/services/collector/raw). The content of the request body is the raw message. The OpenTelemetry Collector supports a simplified version of the raw endpoint, with the ability to accept data from POST requests to the /services/collector/raw path. This is supported out of the box when you define a Splunk HEC receiver endpoint. Here is how you’d typically define such a receiver in your configuration file:     receivers: splunk_hec:     In our case, we want to give it a sensible name. So let’s call it splunk_hec/raw:     receivers: splunk_hec/raw:     From this receiver, we will send the data to a Splunk Enterprise instance. Let’s create that pipeline, and in particular its exporter:     pipelines: logs: receivers: [splunk_hec/raw] processors: [batch] exporters: [splunk_hec/logs] exporters: splunk_hec/logs: token: "00000000-0000-0000-0000-0000000000000" endpoint: "https://splunk:8088/services/collector" source: "output" index: "logs" max_connections: 20 disable_compression: false timeout: 10s insecure_skip_verify: true     This exporter defines the configuration settings of a Splunk HEC endpoint. More documentation and examples are available as part of the OpenTelemetry Collector Contrib github repository. This particular Splunk endpoint says it will send data to the logs index, under the source “output”, to a Splunk instance located under the Splunk hostname, with a HEC token that is just a set of zeroes. This is similar to how we set up in our blog post on log ingestion. Similarly, we set up the Splunk Enterprise instance so it’s ready to accept data. We want to send data periodically to the collector over HTTP. We are going to use Curl to send simple messages to our instance. Here is what a simple curl invocation would look like: curl -XPOST -k localhost:8088/services/collector/raw -d "your message here" -XPOST tells curl to use the HTTP POST method -k removes the need to check for SSL certificates (we’re targeting localhost) localhost:8088/services/collector/raw is the url of the endpoint published by the collector. -d "your message here" is the data being sent as the request body, which will be considered the log message. In our example, we use Docker Compose to create a curl service to periodically send messages to the collector. Here is the curl service:     sender: image: ubuntu:latest command: 'bash -c "apt update && apt install curl -y && while true; do curl -XPOST -k otelcollector:8088/services/collector/raw -d \"$$(date) new message\"; sleep 5; done"' container_name: curl     Our Docker Compose also runs the collector and Splunk Enterprise, similarly to our log ingestion example. We have put this all together into an example that lives under Splunk’s OpenTelemetry Collector github repository. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed. First, check out the repository using git clone: git clone https://github.com/signalfx/splunk-otel-collector.git Using a terminal window, navigate to the folder examples/otel-logs-splunk-raw-hec. Type: docker-compose up This will start the OpenTelemetry Collector, our program generating traces, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display the same information as in our last blog post to let us know it is ready. Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme. You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. Now, we can interrogate our logs index. Enter the search: index=logs to see the logs sent to Splunk:   To send your very own message and see it appear, use the curl command below: curl -XPOST -k localhost:18088/services/collector/raw -d "your message here" When you have finished exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! This concludes our raw log collection example. We have used the OpenTelemetry Collector to successfully ingest raw data and it to our Splunk Enterprise instance. If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository. — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF).The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, or Helm among others. OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open source repository. The repository contains our configuration and hardening parameters as well as examples. This blog post will walk you through using OpenTelemetry processors to route log data to different indexes depending on its content. WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud. This blog post is considered an advanced topic. If you are just catching up with the series, please go back and read on how to set up log ingestion. In this example, we will take a simple log ingestion setup and apply new techniques to change how to route data. Previously, we discussed the concept of pipeline in OpenTelemetry. Pipelines contain 3 element types: receivers, processors, and exporters. We are going to spend a bit more time with processors in this post, but first, we are going to revisit a filelog receiver that is doing more than in our previous post:           receivers: filelog: include: [ /output/file*.log ] operators: - type: regex_parser regex: '(?P<logindex>log\d?)'           What this processor does is reading each line in any of the log files in the output folder starting with file and the log file extension. For each of the lines, it applies a regular expression that reads a portion of the line, extracting log\d? into the attribute logindex. Log records have a body, which is what contains the log message. They can also hold attributes. So each of our records now have an attribute based on the content of the line. Now, we use processors to apply different behaviors based on the value of the logindex attribute:           processors: batch: attributes/log: include: match_type: strict attributes: - { key: logindex, value: 'log' } actions: - key: com.splunk.index action: upsert value: "logs" - key: logindex action: delete           When the value of logindex attribute is ‘log’, do the following: Set the attribute com.splunk.index to the value logs Delete the logindex attribute. The com.splunk.index attribute has significance here as it is used to override the default index configuration value set on the Splunk HEC exporter. You can see all the configuration fields in the README of the Splunk exporter. We are going to create data that will satisfy this processor and route logs to the “logs” index. For fun, we are going to also use the “log2” and “log3” values to route to the “logs2” and “logs3” indexes respectively. Let’s use bash to generate some content as containers in a Docker Compose:           logging: image: bash:latest container_name: logging command: 'bash -c "while(true) do echo \"$$(date) log new message\" >> /output/file.log ; sleep 1; done"' volumes: - ./output:/output logging2: image: bash:latest container_name: logging2 command: 'bash -c "while(true) do echo \"$$(date) log2 new message\" >> /output/file2.log ; sleep 1; done"' volumes: - ./output:/output logging3: image: bash:latest container_name: logging3 command: 'bash -c "while(true) do echo \"$$(date) log3 new message\" >> /output/file3.log ; sleep 1; done"' volumes: - ./output:/output           Our Docker Compose also runs the collector and Splunk Enterprise, similarly to our log ingestion example. We have put this all together into an example that lives under Splunk’s OpenTelemetry Collector github repository. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed. First, check out the repository using git clone: git clone https://github.com/signalfx/splunk-otel-collector.git Using a terminal window, navigate to the folder examples/otel-logs-processor-splunk. Type: docker-compose up This will start the OpenTelemetry Collector, our program generating traces, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display the same information as in our last blog post to let us know it is ready. Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme. You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. Now, we can interrogate our logs index. Enter the search: index=logs to see the logs sent to Splunk: You can change the search to see the contents of the index logs2: When you have finished exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! This concludes our first look at using processors to manipulate log records. We have used the OpenTelemetry Collector to successfully route data to different indexes of our Splunk Enterprise instance. If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository. — Antoine Toulme, Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF) The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, or Helm among others. OpenTelemetry defines a model to represent traces, metrics and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples. This blog post will walk you through sending metrics data with the OpenTelemetry Collector to a Splunk Enterprise deployment. WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud. At our dinner table, I am often asked this question: “How can I send Prometheus metrics data to Splunk Enterprise?” Sometimes, I have to remind my kids to ask me. I also field Harry Potter and Minecraft topics, albeit without as much enthusiasm. In this blog post, we will set up a simple program emitting Prometheus metrics, and we will use the new OpenTelemetry Collector to read those metrics and send them to Splunk. Before we start, I should mention that Splunk Enterprise has a specialized type of index for metrics. If you came here from our previous post on ingesting log data, you will notice that this time we add a bit more information when we define our index in splunk.yml:           splunk: conf: indexes: directory: /opt/splunk/etc/apps/search/local content: metrics: coldPath: $SPLUNK_DB/metrics/colddb datatype: metric homePath: $SPLUNK_DB/metrics/db maxTotalDataSizeMB: 512000 thawedPath: $SPLUNK_DB/metrics/thaweddb           Notice the datatype: metric addition. If your index is not declared as a metric index, you will not be able to ingest metrics through Splunk HEC. To learn more about our metric index technology, please start with our Splunk docs website. Now that we have gotten this out of the way, let’s make sure we have Prometheus metrics to show. Prometheus metrics are exposed on a http endpoint that can be scraped (collected, in OpenTelemetry parlance). We are going to expose a static file that contains valid Prometheus data over http to expose those metrics to our collector. In a Docker compose environment, we would define a container that serves http data.            httpd: image: dragas/thttpd container_name: thttpd restart: always volumes: - ./content:/var/www/http:ro ports: - 80:80           This container is pointing at a content folder. In the folder, we place a file named “metrics”:           # HELP foo_bar The amount of foo going well with bar # TYPE foo_bar gauge foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 3732552.0 foo_bar{application="my-own-app",area="heap",id="G1 Eden Space",} 7.9641776E7 foo_bar{application="my-own-app",area="nonheap",id="Metaspace",} 3.72735E7 foo_bar{application="my-own-app",area="heap",id="G1 Survivor Space",} 2097152.0 foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'non-nmethods'",} 2544904.0 foo_bar{application="my-own-app",area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.0555296E7 foo_bar{application="my-own-app",area="nonheap",id="Compressed Class Space",} 4325676.0 foo_bar{application="my-own-app",area="heap",id="G1 Old Gen",} 5.0331678E7           This file contains JVM metrics related to garbage collection. Now, we’re going to create a metrics pipeline in the collector. This is similar to our last example, with a new receiver named “prometheus”:           pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [splunk_hec/metrics]           The prometheus receiver has a configuration that mirrors the usual Prometheus scrape configuration:           receivers: prometheus: config: scrape_configs: - job_name: 'otel-collector' scrape_interval: 5s static_configs: - targets: ['thttpd:80']           Finally, we set up our Splunk exporter:           exporters: splunk_hec/metrics: # Splunk HTTP Event Collector token. token: "00000000-0000-0000-0000-0000000000000" # URL to a Splunk instance to send data to. endpoint: "https://splunk:8088/services/collector" # Optional Splunk source: https://docs.splunk.com/Splexicon:Source source: "app" # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype sourcetype: "jvm_metrics" # Splunk index, optional name of the Splunk index targeted. index: "metrics" # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100. max_connections: 20 # Whether to disable gzip compression over HTTP. Defaults to false. disable_compression: false # HTTP timeout when sending data. Defaults to 10s. timeout: 10s # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false. # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true. insecure_skip_verify: true           Note that we choose the name splunk_hec/metrics, and we pick the metrics index, but our configuration is the same as the way we set ourselves up to ingest logs otherwise. We have put this all together into an example that lives under Splunk’s OpenTelemetry Collector github repository. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed. First, check out the repository using git clone: git clone https://github.com/signalfx/splunk-otel-collector.git Using a terminal window, navigate to the folder examples/splunk-hec-prometheus. Type: docker-compose up This will start the OpenTelemetry Collector, our bash script generating data, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display the same information as in our last blog post to let us know it is ready. Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme. You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. In this application, click the Analytics Workspace to see our metrics data. Click on the foo_bar metric. You will see a line of data points shown in the graph. In the right-hand panel, click on Split By, and select id. You will now see your data split by its id metric field, as below: You can perform further calculations, such as bucketing values by data or rendering it in a dashboard. When you have finished exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! This concludes our Prometheus metrics example. We have used the OpenTelemetry Collector to successfully scrape Prometheus metrics and send them to our Splunk Enterprise instance. If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository. — Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT ... View more

Introduction This blog post is part of an ongoing series on OpenTelemetry. This guide is addressed to developers, product managers, and enthusiasts worldwide looking to contribute to the OpenTelemetry project. OpenTelemetry is a vast project with the aim of standardizing and promoting observability best practices around metrics, traces, and logs. OpenTelemetry defines a model to represent metrics, traces, and logs. It defines well-known semantic conventions around metric names, dimensions, units, and properties. The project implements the common model into a protobuf model. This model generates code in SDKS associated with different languages. The project also implements a collector which is able to integrate with various technologies, receiving or sending data. The Hero’s Journey As a hopeful contributor to the OpenTelemetry project, you will want to offer changes to a project such as the collector or a specific SDK. Track your work with an issue Your first stop is to check the existing issues on the projects and check if someone has already started working on this. Please create a new issue right away to talk about your work and expose it to the community. It takes time for the community to gel with new ideas - you should be prepared to talk about your idea and your design. If you require help, you can attend a SIG meeting to discuss your proposal. You can also ask questions on the CNCF slack.  When opening an issue, you will be offered a template to help guide you through the steps of qualifying your issue. If you are proposing a new component, you may need a maintainer to sponsor your addition. If you stop there - that is already helping the project move forward! We thank you for your contribution. Shift Left Your Work Devopedia. 2022. "Shift Left." Version 6, February 15. Accessed 2022-06-15. https://devopedia.org/shift-left   The OpenTelemetry project is a bit different from most open source projects as it is not just software - it also defines a specification. As you start working on the project, you will notice that you have to make design choices, such as naming metrics, or making configuration choices and choosing valid defaults. Naming is hard - finding consensus on naming is harder. As you progress, you should open a new issue under the OpenTelemetry specification repository and tie it to your original issue to provide context to maintainers. The specification contains information about SDK environment variables, semantic conventions, sampling, instrumentation, and data serialization methods. Please provide a pull request with your changes, and attend one of the OpenTelemetry Specification SIG meetings to discuss them. Once those changes are merged, you have done the hardest part of contributing to OpenTelemetry, and the design choices that you have made will make the next step easy. However, if you are not a developer, this is already a transformative effort. Thank you for your help! Cultivate the law of Least Surprise   Now armed with a plan and having built consensus around your changes, you are well positioned to implement your changes. A good rule of thumb is to be as least surprising as possible when it’s time to review your code. You should, of course, have great test coverage, and make sure the build passes and that documentation is up to date. You should also prefer small changes instead of a large pull request. It’s much easier to say yes to three small pull requests than a large one. When you open a new pull request against the repository, it will be automatically assigned to a maintainer based on the paths you touch in your pull request. You should leave your pull request as a draft while it is still being developed. An Example I took on the task of adding a new metric to host metrics, the number of threads of a running process as exposed to the hostmetrics receiver. First, I looked for an open issue and found that someone had already filed an issue for this use case. Citing this issue, I created a pull request specifically for this new metric. I joined the specification SIG call to discuss the merits of my new addition: 2022-08-09 meeting I also discussed this addition on the ticket. The specification SIG met one more time to discuss this new metric. Once the specification’s pull request was merged, I could concentrate on the changes to the collector. I changed the hostmetrics receiver to optionally report thread count. I made sure to add this information to the changelog using a yaml file under the unreleased folder at the root of the repository. With this pull request merged, I could start using this new metric after the next release, 0.59. — Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT ... View more

Introduction to Apache Camel Apache Camel is a popular ETL framework with many components. Camel has a long and storied history and is just awesome all around. There is some cool theory behind how it all works: data flows from producers and goes to consumers. Both producers and consumers can be represented as endpoints. The data is manipulated by a set of constructs named enterprise integration patterns. You can buy the reference book about EIPs here, and here is a succinct doc on it on the Camel website. What you really need to take away from this introduction is that if your component is in the list, you can safely assume you can route all sorts of data to it. When I joined Splunk two years ago, I played around one night and sent the Camel team a pull request to add Splunk HEC support. I have to say, my code contribution wasn’t all that good. Thankfully, someone else came around after me and fixed it so it’s just right now. We’re going to use this component today. When you configure Camel, you define routes by which data will transit. In our case, we want to send data to Splunk. To make this interesting, we can reprise the example from Jose where he used the power of Splunk to read and analyze github stats. In our case, we would like to read github releases. Sending Splunk Data In Github releases are available as an Atom feed. I happen to run a Geth node, an Ethereum client, for my day job, so I’d like to know when a new release is available. First, to refer to the Atom feed, I type the endpoint of the feed: https://github.com/ethereum/go-ethereum/releases.atom Since this is going to use the Atom component, I need to point that out by adding a scheme:   atom:https://github.com/ethereum/go-ethereum/releases.atom   The Atom component offers me to split all entries so I don’t read 10 releases at once. This is done by adding splitEntries=true to the query string of the endpoint:   atom:https://github.com/ethereum/go-ethereum/releases.atom?splitEntries=true   Now, we want to send data to Splunk. The endpoint for Splunk is:   splunk-hec:splunk-host/splunk-token?skipTlsVerify=false&index=myindex&source=myindex&sourcetype=mysourcetype&bodyOnly=true   A few notes: Notice the splunk-hec URI scheme. The splunk-host is the host name of the instance. splunk-token is your HEC token. skipTlsVerify sets whether we should skip the verification of the TLS handshake. index, source, sourcetype relate to how your data will be ingested. Those 3 are optional, since they can be defaulted in Splunk itself. bodyOnly is the fix that was contributed to the component, and removes the extra envelope of the Camel message to send the exact message that is present in the body. Apache Camel is easy to configure from there. You just set a “from” and a “to”. It looks like this:   from("atom:https://github.com/ethereum/go-ethereum/releases.atom?splitEntries=true").to("splunk-hec:splunk-host/splunk-token?skipTlsVerify=false&index=myindex&source=myindex&sourcetype=mysourcetype&bodyOnly=true");   This gets you to a working route. Running an Apache Camel Program as a Native Program Two years later I had another few hours to play with Apache Camel, and I tried to see how far I could take the Camel pipeline to make it as easy as possible to run a tool to ingest multiple Atom feeds. I took an unconventional approach, I decided to write my code in Kotlin, allow configuration as a TOML file, and best of all, compile all this to run as a native program. The result is on Github. Please take this for what it is, an experiment to test out a novel packaging method producing a single binary. The program itself is 75 megabytes once compiled down to a native executable. It takes a unique argument of the path of a configuration file. For fun, I created a docker compose environment in the repository you can run to see it in action. It runs Splunk next to it and sends a few chosen Atom feeds of github releases to it: One More Thing The issue with a simple route (from->to) is that we don’t preserve the state of what was consumed. If we stop and restart our route, we might send data to Splunk twice. Camel defines the concept of an idempotent consumer, and offers several implementations. Being a hopeless romantic (I just love how it scales), I have used their Infinispan flavor, tied to a local RocksDB database, to use as a cache here. To set up Infinispan in Kotlin, I simply initiate the cache like so:   val cacheConfig = ConfigurationBuilder().persistence().addStore(RocksDBStoreConfigurationBuilder::class.java) .location(path.resolve("idempotent").toAbsolutePath().toString()) .expiredLocation(path.resolve("idempotent-expired").toAbsolutePath().toString()).build() val infinispanConfig = InfinispanEmbeddedConfiguration() val cacheManager = DefaultCacheManager(GlobalConfigurationBuilder().defaultCacheName("api-scraper").build(), cacheConfig, true) infinispanConfig.cacheContainer = cacheManager val repo = InfinispanEmbeddedIdempotentRepository("atom") repo.configuration = infinispanConfig   Then in-between from and to, we add:   .idempotentConsumer(simple("\${body.id.toASCIIString}"), repo)   simple("\${body.id.toASCIIString}") becomes the key of the entry in the cache. See it in action in a few lines of this Main class. I hope you enjoyed reading this blog post. If you have any questions or comments about the code, feel free to star and open issues on the repository. If you are interested in learning more about Apache Camel, please visit their website and join their mailing lists to get acquainted with their community.  — Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT ... View more

This blog post is part of an ongoing series on OpenTelemetry. Curious about OpenTelemetry but more interested in logs than APM tracing or metrics? Look no further! This blog post will walk you through your first OpenTelemetry Logging pipeline... WARNING: WE ARE DISCUSSING A CURRENTLY UNSUPPORTED CONFIGURATION.  When sending data to Splunk Enterprise, we currently only support the use of the OpenTelemetry Collector in Kubernetes environments. As always, use of the Collector is fully supported to send data to Splunk Observability Cloud. The OpenTelemetry project is the second largest project of the Cloud Native Computing Foundation (CNCF). The CNCF is a member of the Linux Foundation and besides OpenTelemetry, also hosts Kubernetes, Jaeger, Prometheus, and Helm among others. OpenTelemetry defines a model to represent traces, metrics, and logs. Using this model, it orchestrates libraries in different programming languages to allow folks to collect this data. Just as important, the project delivers an executable named the OpenTelemetry Collector, which receives, processes, and exports data as a pipeline. The OpenTelemetry Collector uses a component-based architecture, which allows folks to devise their own distribution by picking and choosing which components they want to support. Please see our official documentation to install the collector. At Splunk, we manage the distribution of our version of the OpenTelemetry collector under this open-source repository. The repository contains our configuration and hardening parameters as well as examples. The OpenTelemetry collector works using a configuration file written in YAML. The collector defines metrics pipelines. For our case, we have defined a pipeline that reads from a file and sends its data to Splunk.       pipelines: logs: receivers: [filelog] processors: [batch] exporters: [splunk_hec/logs]       We use a processor named the batch processor to place multiple entries in one payload, so we avoid sending many messages at once to Splunk. Note we have placed our pipeline under the name logs. That means we intend to use this pipeline to ingest log records. If you have multiple log pipelines, they must start with logs, followed by a slash, and a unique name, such as:       pipelines: logs: … logs/other: … logs/anotherone: …       This notation is also used for other components, such as filelog or splunk_hec in our example. Going back to our pipeline, we have defined it to contain a receiver named "filelog". We have not written it down yet, so let’s write down our receivers:       receivers: filelog: include: [ /output/file.log ]       Our filelog receiver will follow the file /output/file.log and ingest its contents. We also need to define our Splunk HEC exporter. Here is our exporters section:       exporters: splunk_hec/logs: token: "00000000-0000-0000-0000-0000000000000" endpoint: "https://splunk:8088/services/collector" source: "output" index: "logs" max_connections: 20 disable_compression: false timeout: 10s tls: insecure_skip_verify: true       This exporter defines the configuration settings of a Splunk HEC endpoint. More documentation and examples are available as part of the OpenTelemetry Collector Contrib github repository. This particular Splunk endpoint says it will send data to the logs index, under the source “output”, to a Splunk instance located under the Splunk hostname, with a HEC token that is just a set of zeroes. We’re now going to set all the pieces in motion to deliver to you this example end to end. First, we are going to define a program that outputs data to a file.       bash -c "while(true) do echo \"$$(date) new message\" >> /output/file.log ; sleep 1; done"       This bash script will send the current date, accompanied with "new message", every second, until told to stop. Second, we prepare a simple Splunk Enterprise Docker container to run for this example. We set up its logs index with a splunk.yml configuration file:       splunk: conf: indexes: directory: /opt/splunk/etc/apps/search/local content: logs: coldPath: $SPLUNK_DB/logs/colddb datatype: event homePath: $SPLUNK_DB/logs/db maxTotalDataSizeMB: 512000 thawedPath: $SPLUNK_DB/logs/thaweddb       We load up this file by mounting as a volume. We also run the container to set up a default HEC token, open ports, accept the Splunk license, and set a default admin password. Obviously, this is only useful here for our demonstration. There are more interesting configuration possibilities if you follow along this Github repository for Splunk Docker, and be sure to check out Splunk Operator for larger, production-grade deployments. All told, our Splunk server looks like this in our Docker Compose:       # Splunk Enterprise server: splunk: image: splunk/splunk:latest container_name: splunk environment: - SPLUNK_START_ARGS=--accept-license - SPLUNK_HEC_TOKEN=00000000-0000-0000-0000-0000000000000 - SPLUNK_PASSWORD=changeme ports: - 18000:8000 healthcheck: test: ['CMD', 'curl', '-f', 'http://localhost:8000'] interval: 5s timeout: 5s retries: 20 volumes: - ./splunk.yml:/tmp/defaults/default.yml - /opt/splunk/var - /opt/splunk/etc       With that, you are ready to try out our example. To run this example, you will need at least 4 gigabytes of RAM, as well as git and Docker Desktop installed. First, check out the repository using git clone:       git clone https://github.com/signalfx/splunk-otel-collector.git       Using a terminal window, navigate to the folder examples/otel-logs-splunk. Type:       docker-compose up       This will start the OpenTelemetry Collector, our bash script generating data, and Splunk Enterprise. Your terminal will display information as Splunk starts. Eventually, Splunk will display this table to let you know it is available:       splunk | Wednesday 04 May 2022 02:04:18 +0000 (0:00:00.818) 0:00:47.544 ********* splunk | =============================================================================== splunk | splunk_common : Start Splunk via CLI ----------------------------------- 11.97s splunk | splunk_common : Update Splunk directory owner --------------------------- 6.45s splunk | splunk_common : Update /opt/splunk/etc ---------------------------------- 5.42s splunk | splunk_common : Get Splunk status --------------------------------------- 2.50s splunk | Gathering Facts --------------------------------------------------------- 1.67s splunk | splunk_common : Hash the password --------------------------------------- 1.23s splunk | splunk_common : Set options in logs ------------------------------------- 1.11s splunk | splunk_common : Test basic https endpoint ------------------------------- 1.00s splunk | Check for required restarts --------------------------------------------- 0.82s splunk | splunk_standalone : Check for required restarts ------------------------- 0.77s splunk | splunk_standalone : Update HEC token configuration ---------------------- 0.72s splunk | splunk_standalone : Get existing HEC token ------------------------------ 0.69s splunk | splunk_standalone : Setup global HEC ------------------------------------ 0.69s splunk | splunk_common : Check for scloud ---------------------------------------- 0.66s splunk | splunk_common : Generate user-seed.conf (Linux) ------------------------- 0.63s splunk | splunk_common : Find manifests ------------------------------------------ 0.54s splunk | splunk_common : Wait for splunkd management port ------------------------ 0.43s splunk | splunk_common : Cleanup Splunk runtime files ---------------------------- 0.40s splunk | splunk_common : Check for existing splunk secret ------------------------ 0.30s splunk | splunk_common : Check for existing installation ------------------------- 0.30s splunk | =============================================================================== splunk | splunk | Ansible playbook complete, will begin streaming splunkd_stderr.log       Now, you can open your web browser and navigate to http://localhost:18000. You can log in as admin/changeme. You will be met with a few prompts as this is a new Splunk instance. Make sure to read and acknowledge them, and open the default search application. In this application, enter this search to look for logs:       index="logs"       The latest logs generated by the bash script will show: After exploring this example, you can press Ctrl+C to exit from Docker Compose. Thank you for following along! With this example, you have deployed a simple pipeline to ingest the contents of a file into Splunk Enterprise. If you found this example interesting, feel free to star the repository! Just click the star icon in the top right corner. Any ideas or comments? Please open an issue on the repository. — Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT ... View more

Hello! I am an Engineering Manager on the Blockchain & DLT team and I wanted to share a project I recently created. I made a quick map of US States which would highlight my current team’s location. Here is the end result: The challenge was to display multiple states at once. The documentation is helpful when it comes to highlighting just one element on the map:       | stats count | eval featureId="California" | eval count=10000 | geom geo_us_states allFeatures=true       To display multiple results, we will need multiple results. So first off, let’s use makeresults to create results for our query. makeresults has a fun example to create multiple events:       | makeresults | eval test="buttercup rarity tenderhoof dash mcintosh fleetfoot mistmane" | makemv delim=" " test | mvexpand test       We can use that with a list of US states instead:       | makeresults | eval test="California Florida Colorado New Mexico" | makemv delim=" " test | mvexpand test       We rename test to featureId and pass it to geom:       | makeresults | eval featureId="California Florida Colorado New Mexico" | makemv delim=" " featureId | mvexpand featureId |eval count=10000 | geom geo_us_states allFeatures=true       It almost works, but New Mexico is not showing. That’s because we use a space delimiter. Let’s try to separate values with an underscore instead:       | makeresults | eval featureId="California_Florida_Colorado_New Mexico" | makemv delim="_" featureId | mvexpand featureId | eval count=10000 | geom geo_us_states allFeatures=true       Our map now shows values correctly. Was this article helpful to you? Would you like to learn more about SPL techniques? .conf22 is coming up ! Sign up now to join in Las Vegas or remotely. We will be traveling from all the states highlighted on the map to talk about blockchain, so make sure to read about and attend our talk, workshop and NFT activities! — Antoine Toulme,  Senior Engineering Manager, Blockchain & DLT ... View more