Getting Data In

Why am I Unable to fetch Prometheus metrics into Splunk?

TejrajGhadge
Engager

I have used the "Prometheus Metrics for Splunk" plugin from the Splunk Apps to get data from the Prometheus remote write.

Both Prometheus and Splunk are installed on the local Windows machine (for testing). 

A Prometheus remote write is used to send data to the splunk.

Splunk Configuration

````
[prometheusrw]
port = 8098
maxClients = 10

[prometheusrw://856412]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

````

Prometheus configuration

````
- url: "http://localhost:8098"
authorization:
credentials: "ABC123"
tls_config:
insecure_skip_verify: true
write_relabel_configs:
- source_labels: [__name__]
regex: expensive.*
action: drop

````
prometheus error log:

````

ts=2022-07-12T11:40:22.139Z caller=dedupe.go:112 component=remote level=info remote_name=856412 url=http://localhost:8098 msg="Done replaying WAL" duration=10.5184238s
ts=2022-07-12T11:40:22.438Z caller=dedupe.go:112 component=remote level=warn remote_name=856412 url=http://localhost:8098 msg="Failed to send batch, retrying" err="Post \"http://localhost:8098\": EOF"

````

Suggest corrections/ways to get prometheus data to Splunk.

Labels (2)
Tags (2)
0 Karma
1 Solution

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

View solution in original post

luke_monahan
Path Finder

The modular input for Splunk is a binary which is only currently compiled for Linux x86_64. You'll find that in the splunk_modinput_prometheus/bin/ directory.

I have not attempted to compile on Windows to produce a native version here, but it should be possible as Golang is fairly portable. If you would like to have an attempt at compiling a Windows version it may work, otherwise please raise an issue on the Github for native Windows support.

FYI: It definitely works in WSL2 which may solve your problem in the short term.

TejrajGhadge
Engager

@luke_monahan 

Can you please provide more details on how I can use WSL2 to use the plugin?

Tags (1)
0 Karma

luke_monahan
Path Finder

WSL2 will create a Linux environment running as an application under Windows.

You will need to install Splunk in this environment (the same as a normal Linux installation) then install the modular input on that Splunk instance.

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...