Execute a custom command from Splunk Otel receiver

[sagar-yakkundi]

Hi Team,

I was looking to configure the custom command execution like getting the output of ps -ef command or the mq query count.

Can some one please help on how to create a monitoring for the same.

The command which i want to configure are the normal Linux commands which are executed on the server using putty like "ps -ef | grep -i otel" and others

[bishida]

Hi,
For the examples mentioned, I might suggest taking a look at the built-in hostmetrics receiver which you can use to monitor processes like you would with "ps -ef"

https://docs.splunk.com/observability/en/gdi/opentelemetry/components/host-metrics-receiver.html

There are also some available receivers for mq products like ActiveMQ that can provide an mq query count:

https://docs.splunk.com/observability/en/gdi/monitors-messaging/apache-activemq.html#activemq

I can't personally think of an option to invoke a custom command from a receiver, but perhaps another way to consider that goal would be to have a custom command that runs independently of the collector and directs its output to an existing receiver. For example, if your command can generate output in a format that a receiver is listening for, that would be a good way to ingest that metric. Here is an article that discusses that idea:

https://opentelemetry.io/blog/2023/any-metric-receiver/

[sagar-yakkundi]

Hi @bishida 

Sorry , I am new to the Splunk Observability.

Actually we are migrating from one APM solution to Splunk Observability and in the current APM solution, for the majority of application we are monitoring for the below commands

queue printer :  lpstat -o | wc -l
Connection Count RF: netstat -na | grep 49050 |grep -v grep |wc -l
Close Wait RF: netstat -na | grep 49050 |grep -v grep |grep CLOSE_WAIT| wc -l
Close Wait Voice: netstat -na | grep :5200 |grep -v grep |wc -l

These are all individual commands and are running at every a minute. 

We are looking for the same as these are critical to the business and trying to figure out how we can achieve it using Splunk Observability.

Kindly help

[bishida · Accepted solution]

Hi,

It is possible to generate these types of custom command line metrics using the "smartagent/exec" receiver.
https://docs.splunk.com/observability/en/gdi/monitors-databases/exec-input.html

It can be tricky to get the format and approach just right, so here are some tips:
1) Put your command in an external script so it's easier to format the output in an acceptable format and it's also easier to format the call from your receiver. The default format is "influx", so an example of the output you want to generate would look like this:
printerqueue,printer=myprinter length=5

That output would generate a metric named "printerqueue.length" with a value of 5 and a tagname of "printer" and a tagvalue of "myprinter".

Your external script might look like this:

#!/bin/sh

echo printerqueue,printer=myprinter length=$(lpstat -o | wc -l)

2) You'll need to define a receiver in your OTel config (e.g. agent_config.yaml)

receivers:
  smartagent/exec:
    type: telegraf/exec
    command: "/PATH/TO/printerqueue_script.sh"
    telegrafParser:
      dataFormat: "influx"

3) Don't forget to place your new receiver in your metrics pipeline and restart your OTel collector:

service:
  pipelines:
    metrics:
      receivers: [hostmetrics, otlp, signalfx, smartagent/signalfx-forwarder, smartagent/exec]