Getting Data In

Why am I Unable to fetch Prometheus metrics into Splunk?

TejrajGhadge
Engager

I have used the "Prometheus Metrics for Splunk" plugin from the Splunk Apps to get data from the Prometheus remote write.

Both Prometheus and Splunk are installed on the local Windows machine (for testing). 

A Prometheus remote write is used to send data to the splunk.

Splunk Configuration

````
[prometheusrw]
port = 8098
maxClients = 10

[prometheusrw://856412]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

````

Prometheus configuration

````
- url: "http://localhost:8098"
authorization:
credentials: "ABC123"
tls_config:
insecure_skip_verify: true
write_relabel_configs:
- source_labels: [__name__]
regex: expensive.*
action: drop

````
prometheus error log:

````

ts=2022-07-12T11:40:22.139Z caller=dedupe.go:112 component=remote level=info remote_name=856412 url=http://localhost:8098 msg="Done replaying WAL" duration=10.5184238s
ts=2022-07-12T11:40:22.438Z caller=dedupe.go:112 component=remote level=warn remote_name=856412 url=http://localhost:8098 msg="Failed to send batch, retrying" err="Post \"http://localhost:8098\": EOF"

````

Suggest corrections/ways to get prometheus data to Splunk.

Labels (2)
Tags (2)
0 Karma
1 Solution

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

View solution in original post

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

TejrajGhadge
Engager

@luke_monahan 

Can you please provide more details on how I can use WSL2 to use the plugin?

Tags (1)
0 Karma

luke_monahan
Path Finder

WSL2 will create a Linux environment running as an application under Windows.

You will need to install Splunk in this environment (the same as a normal Linux installation) then install the modular input on that Splunk instance.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...