Why am I Unable to fetch Prometheus metrics into Splunk?

[TejrajGhadge]

I have used the "Prometheus Metrics for Splunk" plugin from the Splunk Apps to get data from the Prometheus remote write.

Both Prometheus and Splunk are installed on the local Windows machine (for testing). 

A Prometheus remote write is used to send data to the splunk.

Splunk Configuration

````
[prometheusrw]
port = 8098
maxClients = 10

[prometheusrw://856412]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

````

Prometheus configuration

````
- url: "http://localhost:8098"
authorization:
credentials: "ABC123"
tls_config:
insecure_skip_verify: true
write_relabel_configs:
- source_labels: [__name__]
regex: expensive.*
action: drop

````
prometheus error log:

````

ts=2022-07-12T11:40:22.139Z caller=dedupe.go:112 component=remote level=info remote_name=856412 url=http://localhost:8098 msg="Done replaying WAL" duration=10.5184238s
ts=2022-07-12T11:40:22.438Z caller=dedupe.go:112 component=remote level=warn remote_name=856412 url=http://localhost:8098 msg="Failed to send batch, retrying" err="Post \"http://localhost:8098\": EOF"

````

Suggest corrections/ways to get prometheus data to Splunk.

[luke_monahan · Accepted solution]

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

[TejrajGhadge]

@luke_monahan 

Can you please provide more details on how I can use WSL2 to use the plugin?

[luke_monahan]

WSL2 will create a Linux environment running as an application under Windows.

You will need to install Splunk in this environment (the same as a normal Linux installation) then install the modular input on that Splunk instance.