Prometheus 415 Unsupported Media Type

djmcint · ‎07-28-2021

Followed you Prometheus app setup.

On Heavy forwarder its up and listening on 8098

on Prometheus side seems ok to

However in Prometheus Im been thrown errors on "Unsupported Media Type"

msg="non-recoverable error" count=9 exemplarCount=0 err="server returned HTTP status 415 Unsupported Media Type: <!doctype html><html><head><meta http-equiv=\"content-type\" content=\"text/html; charset=UTF-8\"><title>415 Unsupported Media Type</title></head><body><h1>Unsupported Media Type</h1><p>The requested URL does not support the media type sent.</p></body></html>"

IN prometheus.yml in remote_writes

 - url: "https://myhost.mydomain.com:8089"
   authorization:
     credentials: "ABC123"
   tls_config:
     insecure_skip_verify: true
   write_relabel_configs:
   - source_labels: [__name__]
     regex: expensive.*
     action: drop

in Splunk HF:

[prometheusrw]
port = 8098
maxClients = 10
disabled = 0

[prometheusrw://http_status]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

Help Appreciated

djmcint · ‎07-29-2021

@luke_monahan effectively a typo . Thanks again, works like a charm

luke_monahan · ‎07-28-2021

Your Prometheus config is sending the data to port 8089 -- the main Splunkd port, rather than the prometheusrw port.

The prometheusrw input opens up a new port (8098) that can handle the content-type that Prometheus will send. You need to update your Prometheus configuration to use port 8098 and (in this case) use http and not https.

Prometheus 415 Unsupported Media Type

configuration

Extending Observability Content to Splunk Cloud

More Control Over Your Monitoring Costs with Archived Metrics!

New in Observability Cloud - Explicit Bucket Histograms