Getting Data In

Why am I Unable to fetch Prometheus metrics into Splunk?

TejrajGhadge
Engager

I have used the "Prometheus Metrics for Splunk" plugin from the Splunk Apps to get data from the Prometheus remote write.

Both Prometheus and Splunk are installed on the local Windows machine (for testing). 

A Prometheus remote write is used to send data to the splunk.

Splunk Configuration

````
[prometheusrw]
port = 8098
maxClients = 10

[prometheusrw://856412]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

````

Prometheus configuration

````
- url: "http://localhost:8098"
authorization:
credentials: "ABC123"
tls_config:
insecure_skip_verify: true
write_relabel_configs:
- source_labels: [__name__]
regex: expensive.*
action: drop

````
prometheus error log:

````

ts=2022-07-12T11:40:22.139Z caller=dedupe.go:112 component=remote level=info remote_name=856412 url=http://localhost:8098 msg="Done replaying WAL" duration=10.5184238s
ts=2022-07-12T11:40:22.438Z caller=dedupe.go:112 component=remote level=warn remote_name=856412 url=http://localhost:8098 msg="Failed to send batch, retrying" err="Post \"http://localhost:8098\": EOF"

````

Suggest corrections/ways to get prometheus data to Splunk.

Labels (2)
Tags (2)
0 Karma
1 Solution

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

View solution in original post

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

TejrajGhadge
Engager

@luke_monahan 

Can you please provide more details on how I can use WSL2 to use the plugin?

Tags (1)
0 Karma

luke_monahan
Path Finder

WSL2 will create a Linux environment running as an application under Windows.

You will need to install Splunk in this environment (the same as a normal Linux installation) then install the modular input on that Splunk instance.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...