I am a beginner with Splunk.
I am setting up Splunk Enterprise in a three-tier architecture with a Search Head server, an Indexer server, and a Heavy Forwarder server. I want to install the Splunk Add-on for Microsoft Cloud Services on the Heavy Forwarder server to ingest data from Azure Event Hubs.
However, when I check the logs of the installed add-on, I see the following error:
(splunk_ta_microsoft-cloudservices_azure_audit.log)
splunk_ta_microsoft-cloudservices_azure_audit.log
2024-12-13 02:44:48,835 +0000 log_level=ERROR, pid=33699, tid=MainThread, file=rest.py, func_name=splunkd_request, code_line_no=67 | Failed to send rest request=https://127.0.0.1:8089/services/server/info, errcode=unknown, reason=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 175, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 723, in urlopen
chunked=chunked,
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 1061, in _validate_conn
conn.connect()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 187, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f48c2a95e50>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
~~~
Concern Point #1
It seems that the error has been resolved by adding the following line to
/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local/web.conf
(just changing the request destination from
<local of the Heavy Forwarder server>
to
<IP address of the Search Head server>
)
[settings]
mgmtHostPort = <IP address of the Search Head server>:8089
However, I am now seeing the following log, and a 401 is being returned. The request destination is
https://127.0.0.1:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs_settings?count=-1
Concern Point #2
I thought I could resolve Concern Point #1 in the same way by changing the request destination to the
<IP address of the Search Head server>
, but I don't know how to do that (I'm unsure if this approach is correct, so I would appreciate your guidance).
splunk_ta_microsoft-cloudservices_azure_audit.log
2024-12-13 10:41:22,011 +0000 log_level=ERROR, pid=194872, tid=MainThread, file=config.py, func_name=log, code_line_no=66 | UCC Config Module: Fail to load endpoint "global_settings" - Unspecified internal server error. reason={"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [401]: Unauthorized -- call not properly authenticated\". See splunkd.log/python.log for more details."}]}
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_azure_audit.py", line 21, in <module>
schema_para_list=("description",),
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_mod_input.py", line 232, in main
log_suffix=log_suffix,
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_mod_input.py", line 130, in run
tconfig = tc.create_ta_config(settings, config_cls or tc.TaConfig, log_suffix)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 228, in create_ta_config
return config_cls(meta_config, settings, stanza_name, log_suffix)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 53, in __init__
self._load_task_configs()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 75, in _load_task_configs
config_handler = th.ConfigSchemaHandler(self._meta_config, self._client_schema)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_helper.py", line 95, in __init__
self._load_conf_contents()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_helper.py", line 120, in _load_conf_contents
self._all_conf_contents = self._config.load()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/config.py", line 143, in load
log(msg, level=logging.ERROR, need_tb=True)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/config.py", line 64, in log
stack = "".join(traceback.format_stack())
NoneType: None
~~~
Supplementary Information
The results of `curl` commands on the Heavy Forwarder server are as follows:
If you need any further adjustments or specific aspects to focus on, please let me know!
