Ingesting Protobuf Messages - OpenTelementry ProtoBuf Data

[senthilec566]

Hello Team,

We have a requirement to support Protobuf data ingestion for Splunk Endpoint. Many customers have expressed interest in sending data to Splunk in Protobuf Messages and making it available for search.

What's the input?
https://github.com/open-telemetry/opentelemetry-proto/blob/v1.0.0/opentelemetry/proto/collector/logs...

The input would be the ProtoBuf Message: ExportLogsServiceRequest

unmarshalled proto
[
resource:{attributes:{key:"cloud.provider" value:{string_value:"data"}} attributes:{key:"ew_id" value:{string_value:"3421"}} attributes:{key:"ip" value:{string_value:"0.1.0.1"}}} scope_logs:{log_records:{time_unix_nano:1714188733 observed_time_unix_nano:1714188733 severity_text:"FATAL" body:{string_value:"onOriginRequest%20error%20level%2065553GXK3l7A1TG7QNiNsif0M4eZ7RmimyGeSu8GfyjGQTmbxjOEpDktybtjuWpb"} attributes:{key:"requestId" value:{string_value:"123456 Fp5zWvbr2cdYaOgC2LmC7hEs2"}} attributes:{key:"custom" value:{string_value:"3421 LUl8ovNHb6jO9Ak"}} attributes:{key:"queueit" value:{string_value:"1.2.3 sWcAL"}} attributes:{key:"ds2custom_message" value:{string_value:"Splunk POC Request 3qE2lAUxf0iDyCcxeNZkra3gK"}} trace_id:"\xd3\xcd8\xd3m5\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4" span_id:"ӽ7\xd3m5\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4\xd7]u"}} ]

curl -k -vvv -H "Authorization: Splunk XXXXX" -H 'Content-Type: application/x-protobuf' 'https://prd-p-pwf16.splunkcloud.com:8088/services/collector' --data-binary @data


How to ingest the probuf message?

[marnall]

You might find this blog post useful:

https://www.splunk.com/en_us/blog/tips-and-tricks/protocol-data-inputs.html

It describes the Protocol Data Inputs app (https://splunkbase.splunk.com/app/1901) that performs custom data handling and pre-processing of the received data before it gets indexed by Splunk.

It should be possible with this app to write a custom data handler that will accept your ProtoBuf data.

[tscroggins]

Hi @senthilec566,

You can't send a protobuf message directly to the HTTP Event Collector service. If you're working with an application you've developed, you may find what you need in Splunk OpenTelemetry Collector at https://github.com/signalfx/splunk-otel-collector and its splunk_hec exporter.

There are no currently maintained OTel or protobuf modular inputs, but you may enjoy building or reusing a solution from Vert.x under the Protocol Data Inputs add-on at https://splunkbase.splunk.com/app/1901 . Vert.x provides many modules at https://vertx.io.

I've also provided a bespoke protobuf example in the past at https://community.splunk.com/t5/All-Apps-and-Add-ons/Could-Splunk-ingestion-proto-buff-msg-via-HEC-e....