topic Re: Configure Prometheus with Splunk in Deployment Architecture

Configure Prometheus with Splunk

ShubhneetChawla — Thu, 11 Jun 2020 16:34:54 GMT

Hi ,

I am a developer and looking to integrate my Prometheus with splunk
Prometheus is up and running at my local (http://localhost:9090)

Objective is to post metrics data(Hyperledger fabric metrics) from Prometheus directly to splunk for monitoring, instead of Grafana
For grafana i just need to provide the Data source to it ie. (http://localhost:9090)

But i am unable to do the same for splunk

Steps i followed :

1. Created a HTTP event listener (HEC) on splunk.

2. Fetched the Token

3. Passed the Token and url in Prometheus.yaml

Prometheus_File_Changes

4. When i start the Prometheus container for establishing connection with splunk It gives me below error

Error

Kindly guide me how to post my metrics/ Prometheus URL to splunk such that that i can monitor my metrics

Do let me know if i miss onto any step .

@splunk #Prometheus #metrics #HyperledgerFabricV2.0 #metrics #Hyper-ledger @Anonymous #IBM

Re: Configure Prometheus with Splunk

twesty — Thu, 11 Jun 2020 20:57:22 GMT

it looks like you may be trying to run before you can walk.

It looks like you have setup your HEC endpoint, however the first thing I would do is look at trying to send a test message to it from the same box using a script.

The error also indicates it could be to do with the content sent through. Either the message could be too large (fairly common with xml, which this looks like it is) or there's something within. if possible I would look to try and restructure the data sent through to Splunk so the message is easier to work with when inside of Splunk.

Re: Configure Prometheus with Splunk

ShubhneetChawla — Fri, 12 Jun 2020 04:31:36 GMT

Indeed ,

I am very much new to splunk and its processes.

Well yes the Data is quite large and of real time (statsD, metrics, healthz), Though i successfully send data via Curl commands that particular HEC.

It would be good if I could connect to for guidance.

Re: Configure Prometheus with Splunk

ShubhneetChawla — Fri, 12 Jun 2020 04:32:29 GMT

Can i Connect to you on hangout or something.

Thanks in Advance

Re: Configure Prometheus with Splunk

twesty — Fri, 12 Jun 2020 08:38:22 GMT

Sorry I dont think there much else I would be able to assist with. I would suggest that if you need further guidance, to speak to your local account manager and support team who may be able to speak to you in further detail.

Your other option is to reach out to a partner and see what they can offer.