All Apps and Add-ons

Ingesting Protobuf Messages - OpenTelementry ProtoBuf Data

senthilec566
New Member

Hello Team,

We have a requirement to support Protobuf data ingestion for Splunk Endpoint. Many customers have expressed interest in sending data to Splunk in Protobuf Messages and making it available for search.


What's the input?
https://github.com/open-telemetry/opentelemetry-proto/blob/v1.0.0/opentelemetry/proto/collector/logs...

The input would be the ProtoBuf Message: ExportLogsServiceRequest

unmarshalled proto
[
resource:{attributes:{key:"cloud.provider" value:{string_value:"data"}} attributes:{key:"ew_id" value:{string_value:"3421"}} attributes:{key:"ip" value:{string_value:"0.1.0.1"}}} scope_logs:{log_records:{time_unix_nano:1714188733 observed_time_unix_nano:1714188733 severity_text:"FATAL" body:{string_value:"onOriginRequest%20error%20level%2065553GXK3l7A1TG7QNiNsif0M4eZ7RmimyGeSu8GfyjGQTmbxjOEpDktybtjuWpb"} attributes:{key:"requestId" value:{string_value:"123456 Fp5zWvbr2cdYaOgC2LmC7hEs2"}} attributes:{key:"custom" value:{string_value:"3421 LUl8ovNHb6jO9Ak"}} attributes:{key:"queueit" value:{string_value:"1.2.3 sWcAL"}} attributes:{key:"ds2custom_message" value:{string_value:"Splunk POC Request 3qE2lAUxf0iDyCcxeNZkra3gK"}} trace_id:"\xd3\xcd8\xd3m5\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4" span_id:"ӽ7\xd3m5\xd3M4\xd3M4\xd3M4\xd3M4\xd3M4\xd7]u"}} ]

 

curl -k -vvv -H "Authorization: Splunk XXXXX" -H 'Content-Type: application/x-protobuf' 'https://prd-p-pwf16.splunkcloud.com:8088/services/collector' --data-binary @data


How to ingest the probuf message?

0 Karma

marnall
Motivator

You might find this blog post useful:

https://www.splunk.com/en_us/blog/tips-and-tricks/protocol-data-inputs.html

It describes the Protocol Data Inputs app (https://splunkbase.splunk.com/app/1901) that performs custom data handling and pre-processing of the received data before it gets indexed by Splunk.

It should be possible with this app to write a custom data handler that will accept your ProtoBuf data.

0 Karma

tscroggins
Influencer

Hi @senthilec566,

You can't send a protobuf message directly to the HTTP Event Collector service. If you're working with an application you've developed, you may find what you need in Splunk OpenTelemetry Collector at https://github.com/signalfx/splunk-otel-collector and its splunk_hec exporter.

There are no currently maintained OTel or protobuf modular inputs, but you may enjoy building or reusing a solution from Vert.x under the Protocol Data Inputs add-on at https://splunkbase.splunk.com/app/1901 . Vert.x provides many modules at https://vertx.io.

I've also provided a bespoke protobuf example in the past at https://community.splunk.com/t5/All-Apps-and-Add-ons/Could-Splunk-ingestion-proto-buff-msg-via-HEC-e....

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Unlock What’s Next: The Splunk Cloud Platform at .conf25

In just a few days, Boston will be buzzing as the Splunk team and thousands of community members come together ...