Getting Data In

Is it possible to write a lightweight custom forwarder to collect data, and not have to deploy the universal forwarder on every machine that needs monitoring?

sbroberg
Engager

We're trying to determine if Splunk is appropriate for our scenario, which is to monitor our own agent that runs on our users' PCs and Macs. We have several million customers, and it seems like it would be burdensome (based on the posted system requirements) to deploy a universal forwarder onto every user's machine (plus I'm not sure how we would integrate this into the existing installer & upgrader features of our app).

All we really need to do is to periodically upload (either daily or hourly) a .json file containing some structured data for metrics that describe the current state of the app during that interval, as well as some exception events (crashes, thrown exceptions of note, etc.). In theory, this would just be an HTTPS call to our Splunk instance with the appropriate authentication, but I can't locate any online documentation that describes this - the REST API seems to be more about controlling existing collectors and doing extraction & analysis of collected data.

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey,

as far as my experience goes, the Universal Forwarder is not really ressource intense, however there is an option that fits so well that it feels as it had only been made for your question. 😉

Check out the Splunk HTTP Event Collector. There is a lot of documentation that allows to send data via HTTP, control authentication and other stuff.
If you're Python-literate, you could take a look at this class written by George Starcher, it's really fast and easily transmits large amounts of data to Splunk: Splunk-Class-httpevent

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

This sounds like the perfect case for the HTTP Event Collector (HEC). The HEC reads JSON-encoded events sent via HTTP(S). The universal forwarder is not needed. See http://docs.splunk.com/Documentation/Splunk/7.1.1/Data/HECWalkthrough.

---
If this reply helps you, Karma would be appreciated.
0 Karma

xpac
SplunkTrust
SplunkTrust

Hey,

as far as my experience goes, the Universal Forwarder is not really ressource intense, however there is an option that fits so well that it feels as it had only been made for your question. 😉

Check out the Splunk HTTP Event Collector. There is a lot of documentation that allows to send data via HTTP, control authentication and other stuff.
If you're Python-literate, you could take a look at this class written by George Starcher, it's really fast and easily transmits large amounts of data to Splunk: Splunk-Class-httpevent

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...