Instrumenting metrics in a C++ application

[MisterBoy]

Hi, I'm brand new to Splunk coming from a background using Prometheus metrics.

I've been reading through Splunk docs but I'm struggling to get some of the basics down. I was looking at https://docs.splunk.com/Documentation/Splunk/8.1.1/Metrics/GetMetricsInOther which talks about HTTP/JSON but is this saying my application would offer an HTTP endpoint for polling/pulling, or my application needs to call an endpoint on the Splunk server?

I've always worked using the poll paradigm before and one thing I can't seem to find are examples of how to instrument metrics in my own code. In particular, 90% of our system (which comprises dozens of modules) is written in C++ and I would really love to see a code example of instrumenting and exposing a metric (e.g. number of calls to MyMethod() in C++, for consumption by Splunk Enterprise.

Many thanks for any help.

[richgalloway]

Your application needs to send data to an HTTP endpoint on a Splunk server.

How to instrument your code perhaps is a better question for StackOverflow. One approach, however, is to call a function whenever something you want to measure happens. That function will increment the appropriate static variable. A separate thread initiates a timer and, when the timer pops, collects the measurement variables into a JSON string, sends it to Splunk, then resets the variables. Implementation details are left as an exercise for the reader.

[MisterBoy]

Thanks @richgalloway . Does Splunk provide any libraries for widely used languages? Measurement code needs to be very performant and thread-safe so this is not a trivial task, especially if either Splunk or the community has already created such things that are tried and tested.

And Splunk is exclusively a push-based metrics store, just to be absolutely double-sure on that?

[richgalloway]

Splunk provides a few SDKs: Java, Python, and C#, if memory serves.  C++ is not on the list.

Splunk is exclusively push-based, with a few exceptions that don't apply here (like querying an SQL database).