Getting Data In

Is it possible to write a lightweight custom forwarder to collect data, and not have to deploy the universal forwarder on every machine that needs monitoring?

sbroberg
Engager

We're trying to determine if Splunk is appropriate for our scenario, which is to monitor our own agent that runs on our users' PCs and Macs. We have several million customers, and it seems like it would be burdensome (based on the posted system requirements) to deploy a universal forwarder onto every user's machine (plus I'm not sure how we would integrate this into the existing installer & upgrader features of our app).

All we really need to do is to periodically upload (either daily or hourly) a .json file containing some structured data for metrics that describe the current state of the app during that interval, as well as some exception events (crashes, thrown exceptions of note, etc.). In theory, this would just be an HTTPS call to our Splunk instance with the appropriate authentication, but I can't locate any online documentation that describes this - the REST API seems to be more about controlling existing collectors and doing extraction & analysis of collected data.

0 Karma
1 Solution

xpac
SplunkTrust
SplunkTrust

Hey,

as far as my experience goes, the Universal Forwarder is not really ressource intense, however there is an option that fits so well that it feels as it had only been made for your question. 😉

Check out the Splunk HTTP Event Collector. There is a lot of documentation that allows to send data via HTTP, control authentication and other stuff.
If you're Python-literate, you could take a look at this class written by George Starcher, it's really fast and easily transmits large amounts of data to Splunk: Splunk-Class-httpevent

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

This sounds like the perfect case for the HTTP Event Collector (HEC). The HEC reads JSON-encoded events sent via HTTP(S). The universal forwarder is not needed. See http://docs.splunk.com/Documentation/Splunk/7.1.1/Data/HECWalkthrough.

---
If this reply helps you, Karma would be appreciated.
0 Karma

xpac
SplunkTrust
SplunkTrust

Hey,

as far as my experience goes, the Universal Forwarder is not really ressource intense, however there is an option that fits so well that it feels as it had only been made for your question. 😉

Check out the Splunk HTTP Event Collector. There is a lot of documentation that allows to send data via HTTP, control authentication and other stuff.
If you're Python-literate, you could take a look at this class written by George Starcher, it's really fast and easily transmits large amounts of data to Splunk: Splunk-Class-httpevent

Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂

Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...