Cannot Ingest Prometheus Data: inputs.conf - recieving errors: btool does not list the stanza [prometheusrw])

[Network007]

Hello Splunk Community, 

I'm encountering an issue with ingesting data from a Prometheus remote_write_agent into Splunk Enterprise – this solution utilises the ‘Prometheus Metrics for Splunk and is within a Test Environment.

Problem Summary: Despite ensuring that the 'inputs.conf' file matches the configuration specifications defined in the 'inputs.conf.spec' file, the Prometheus data is not being ingested and I am receiving errors, e.g port: Not found in "btool" output (btool does not list the stanza [prometheusrw]) when viewing the inputs.conf file in the config explorer application.

Details:

Splunk Version: Splunk Enterprise 9.2 (Trial License)

Operating System: Ubuntu 22.04

Splunk Application: Prometheus Metrics for Splunk (Latest Version 1.0.1)

inputs.conf.spec

 /opt/splunk/etc/apps/modinput_prometheus/README/inputs.conf.spec

(Full inputs.conf.spec - https://github.com/lukemonahan/splunk_modinput_prometheus/blob/master/modinput_prometheus/README/inp...

As seen in image, the inputs.conf.spec file states there is a port  and maxClients configuration parameters.

Network007_0-1718958600783.png

In the inputs.conf I updated the  /opt/splunk/etc/apps/modinput_prometheus/local/inputs.conf file to include the details below which meet the required formatting above:

Network007_1-1718958660305.png

The inputs.conf file was saved, and the Splunk Server rebooted. After rebooting the input.conf was checked to ensure the config specification where being accepted using the Config Explorer App – 

These errors where received for the following configuration parameters:

Network007_2-1718958697322.png

Network007_3-1718958712578.pngNetwork007_4-1718958722647.png

However, other configuration parameters such as index, sourcetype & whitelist 
Returned: 'Found in "btool" output. Exists in spec file (Stanza=[prometheusrw]) - and were accepted by Splunk.

Network007_6-1718958848185.png

For some unknown reason, Splunk is not recognising some of the configuration parameters above that are listed within the inputs.conf.spec file, even when formatted accordingly.

Other Information:

Prometheus remote-write-exporter details:

Network007_7-1718958974448.png

Splunk Index: skyline_prometheus_metrics

Network007_8-1718959016503.png

Any assistance is appreciated, thank you Splunk Community 🙂