Build Your Very own Splunk Discord bot with OpenTelemetry

[atoulme]

This blog post is part of an ongoing series on OpenTelemetry.

This blog post shows how you can use the OpenTelemetry Collector, a Discord bot library, and create a cool solution to follow Discord activity or post alerts. Discord is a free SaaS forum software used by cool kids to communicate. And OpenTelemetry is a great way to collect traces, metrics, and logs.

To get started, we create a bot following this tutorial which explains how to create a simple Discord bot in Golang. There are two parts to this example. First, you create an application in Discord using the developers portal and a bot associated with it. Second, you write just enough go code to make the bot reply pong when it sees the message "ping".

Taking this example as inspiration, we can create our own bot that is able to handle messages for starters.

func (b *botImpl) Start() error {
  log.Info().Msg("Starting bot")
  goBot, err := discordgo.New("Bot " + b.token)
  if err != nil {
     return err
  }
  goBot.AddHandler(b.messageHandler)
  err = goBot.Open()
  if err != nil {
     return err
  }
  log.Info().Msg("Started bot")
  b.goBot = goBot
  return nil
}

This bot defines a message handler function, messageHandler:

func (b *botImpl) messageHandler(s *discordgo.Session, m *discordgo.MessageCreate) {
  data, err := json.Marshal(m)
  if err != nil {
     log.Error().Err(err).Msg("Error marshaling message")
     return
  }
  b.sink(m.Timestamp, data)
}

This call to b.sink calls out to a function passed into the bot:

sink  func(timestamp time.Time, data []byte)

That's it for our bot for now.

The sink function itself calls out to a "hecClient":

func(timestamp time.Time, bytes []byte) {
  err := hecClient.SendData(timestamp, bytes)
  if err != nil {
     logger.Error("Error sending data", zap.Error(err))
  }
}

The HEC client is a simple wrapper for the Splunk HEC exporter component from OpenTelemetry.
We create and start our exporter programmatically like so:

exporter, err := factory.CreateLogsExporter(context.Background(), component.ExporterCreateSettings{
  TelemetrySettings: component.TelemetrySettings{
     Logger:         logger,
     TracerProvider: trace.NewNoopTracerProvider(),
     MeterProvider:  nonrecording.NewNoopMeterProvider(),
     MetricsLevel:   configtelemetry.LevelNone,
  },
  BuildInfo: component.NewDefaultBuildInfo(),
}, hecConfig)
if err != nil {
  return nil, err
}
if err = exporter.Start(context.Background(), componenttest.NewNopHost()); err != nil {
  return nil, err
}

We wrap the exporter with a batch processor so we'll send multiple events at once if possible:

batchProcessorFactory := batchprocessor.NewFactory()
processor, err := batchProcessorFactory.CreateLogsProcessor(context.Background(), componenttest.NewNopProcessorCreateSettings(), batchProcessorFactory.CreateDefaultConfig(), exporter)

if err != nil {
  return nil, err
}

if err = processor.Start(context.Background(), componenttest.NewNopHost()); err != nil {
  return nil, err
}

Since we're outside the collector, we stub some of the telemetry and configuration settings available.

To test out this bot, we create a Discord application and create a bot there with a unique ID that we drop in our configuration.

We invite our bot to a channel of our choice and send messages to it - we can see the messages show up in Splunk in real-time as the bot is able to read them.

The messages are quite comprehensive as they contain author information and metadata.

One more thing - since our bot can listen for messages, it can send them as well. We create a HTTP server exposed by the bot that can process webhook actions from Splunk alerts.

listen := s.listenAddr
if listen == "" {
  listen = httpAddr
}

s.webServer = &http.Server{
  Addr:         listen,
  Handler:      s,
  ReadTimeout:  serverReadTimeout,
  WriteTimeout: serverWriteTimeout,
  IdleTimeout:  serverIdleTimeout,
}

err := s.webServer.ListenAndServe()
if err != http.ErrServerClosed {
  return err
}

return nil

The server takes a handler method that processes webhooks and posts to the channel ID associated, if found:

wh := req.URL.Query().Get("webhook")

var cfg *config.WebhookConfig

for _, whc := range s.webhooks {
  if whc.ID == wh {
     cfg = whc
     break
  }
}

if cfg == nil {
  resp.WriteHeader(404)
  s.logger.Debug("no webhook defined", zap.String("remoteAddr", req.RemoteAddr), zap.String("webhook", wh))
  return

}



var alertRequest AlertRequest

err := json.NewDecoder(req.Body).Decode(&alertRequest)
if err != nil {
  s.logger.Debug("bad request", zap.String("remoteAddr", req.RemoteAddr), zap.Error(err))
  resp.WriteHeader(400)
  return
}

err = s.bot.SendMessage(cfg.Channel, fmt.Sprintf("%s - see results %s", alertRequest.SearchName, alertRequest.ResultsLink))

if err != nil {
  s.logger.Error("error sending message to Discord", zap.String("remoteAddr", req.RemoteAddr), zap.Error(err))
  resp.WriteHeader(500)
  return
}

resp.WriteHeader(200)

The  webhook config comes from our configuration file:

{
 "token": "REPLACE WITH DISCORD BOT TOKEN",
 "hec_endpoint": "http://localhost:8808/collector/event",
 "hec_token": "00000000-0000-0000-0000-0000000000000",
 "hec_index": "main",
 "hec_insecure_skip_verify": true,
 "listen_addr": "0.0.0.0:8080"
 "webhooks": [
   {
     "id": "foo",
     "channel": "REPLACE WITH CHANNEL ID"
   }
 ]
}

Note the "foo" id.

Now in Splunk, we define an alert that will call the webhook foo. We create a real-time alert reacting to a search for the term "boo". The alert will call out to a webhook with http://bot:8080/?webhook=foo.

To put this together, we can try to type "boo" in Discord - this message gets ingested in Splunk, which reacts by posting an alert to our Discord channel:

That’s all folks! The code is available under Apache 2.0 License for your sheer enjoyment. Your patches are very welcome.

— Antoine Toulme, Senior Engineering Manager, Blockchain & DLT