Splunk otel solr configuration with user name and password

[Elango]

We are setting up the splunk otel collector in ssl and authorization enabled solr. We are facing issue in passing the username and password for solr endpoint in the agent_config.yaml file.  Refer the bbelow content of the config file, due to security reason, we have masked the hostname, userid, password details.  

receivers:

  smartagent/solr:
  type: collectd/solr
  host: <hostname>
  port: 6010
  enhancedMetrics: true
 

exporters:
sapm:
access_token: "${SPLUNK_ACCESS_TOKEN}"
endpoint: "${SPLUNK_TRACE_URL}"
signalfx:
access_token: "${SPLUNK_ACCESS_TOKEN}"
api_url: "${SPLUNK_API_URL}"
ingest_url: "${SPLUNK_INGEST_URL}"
sync_host_metadata: true
headers:
username: <username>
password: <password>
correlation:
otlp
tls:
insecure: false
cert_file: <certificate_file>.crt
key_file: <key_file>.key

Error Log : 

-- Logs begin at Fri 2023-11-17 23:32:38 EST, end at Tue 2023-11-28 02:46:22 EST. --
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/site-packages/sfxrunner/scheduler/simple.py", line 57, in _call_on_interval
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: func()
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/collectd-python/solr/solr_collectd.py", line 194, in read_metrics
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: solr_cloud = fetch_collections_info(data)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/collectd-python/solr/solr_collectd.py", line 328, in fetch_collections_info
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: get_data = _api_call(url, data["opener"])
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/collectd-python/solr/solr_collectd.py", line 286, in _api_call
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: resp = urllib.request.urlopen(req)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 216, in urlopen
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: return opener.open(url, data, timeout)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 519, in open
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: response = self._open(req, data)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 536, in _open
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: result = self._call_chain(self.handle_open, protocol, protocol +
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 496, in _call_chain
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: result = func(*args)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 1377, in http_open
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: return self.do_open(http.client.HTTPConnection, req)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/urllib/request.py", line 1352, in do_open
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: r = h.getresponse()
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/http/client.py", line 1378, in getresponse
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: response.begin()
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/http/client.py", line 318, in begin
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: version, status, reason = self._read_status()
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: ^^^^^^^^^^^^^^^^^^^
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: File "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/http/client.py", line 300, in _read_status
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: raise BadStatusLine(line)
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: [30B blob data]
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: [3B blob data]
Nov 18 14:06:48 aescrsbsql01.scr.dnb.net otelcol[1035]: {"kind": "receiver", "name": "smartagent/solr", "data_type": "metrics", "createdTime": 1700334408.8198304, "lineno": 56, "logger": "root", "monitorID": "smartagentsolr", "monitorType": "collectd/solr", "runnerPID": 1703, "sourcePath": "/usr/lib/splunk-otel-collector/agent-bundle/lib/python3.11/site-packages/sfxrunner/logs.py"}
Nov 18 14:06:58 aescrsbsql01.scr.dnb.net otelcol[1035]: 2023-11-18T14:06:58.821-0500 error signalfx/handler.go:188 Traceback (most recent call last):

[bishida]

Hi,

I don't have a solr instance to test, but here are some thoughts that might help you troubleshoot.

https://docs.splunk.com/observability/en/gdi/monitors-hosts/solr.html
https://github.com/signalfx/collectd-solr

I'm not sure that this receiver supports basic auth with username and password. It looks like it might support certificate based authentication, so that is something you might want to try.

In the sample you shared, you probably don't want any of the configuration in the "exporters" section. Anything you try with configuration of the solr receiver will be in the "receivers" section. Also, don't forget to put the receiver in your metrics pipeline later in the config file.

The sample also contains some unconventional indentation--please remember that the yaml configs are indentation sensitive--so you'll need to have every line indented with spaces just right.

[Elango]

Thanks a lot for the response, I tried multiple options, but none of them is working.