Configuring Splunk Add-on for Microsoft Cloud Services with 3 tiers Splunk Enterprise

[km]

I am a beginner with Splunk.

I am setting up Splunk Enterprise in a three-tier architecture with a Search Head server, an Indexer server, and a Heavy Forwarder server. I want to install the Splunk Add-on for Microsoft Cloud Services on the Heavy Forwarder server to ingest data from Azure Event Hubs.

However, when I check the logs of the installed add-on, I see the following error:
(splunk_ta_microsoft-cloudservices_azure_audit.log)

splunk_ta_microsoft-cloudservices_azure_audit.log
2024-12-13 02:44:48,835 +0000 log_level=ERROR, pid=33699, tid=MainThread, file=rest.py, func_name=splunkd_request, code_line_no=67 | Failed to send rest request=https://127.0.0.1:8089/services/server/info, errcode=unknown, reason=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 175, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 723, in urlopen
chunked=chunked,
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 404, in _make_request
self._validate_conn(conn)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connectionpool.py", line 1061, in _validate_conn
conn.connect()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/urllib3/connection.py", line 187, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f48c2a95e50>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:
~~~

Concern Point #1
It seems that the error has been resolved by adding the following line to

/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/local/web.conf

 (just changing the request destination from

<local of the Heavy Forwarder server>​

to

<IP address of the Search Head server>

)

[settings]
mgmtHostPort = <IP address of the Search Head server>:8089

However, I am now seeing the following log, and a 401 is being returned. The request destination is

https://127.0.0.1:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs_settings?count=-1

Concern Point #2
I thought I could resolve Concern Point #1 in the same way by changing the request destination to the

<IP address of the Search Head server>

, but I don't know how to do that (I'm unsure if this approach is correct, so I would appreciate your guidance).

splunk_ta_microsoft-cloudservices_azure_audit.log
2024-12-13 10:41:22,011 +0000 log_level=ERROR, pid=194872, tid=MainThread, file=config.py, func_name=log, code_line_no=66 | UCC Config Module: Fail to load endpoint "global_settings" - Unspecified internal server error. reason={"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [401]: Unauthorized -- call not properly authenticated\". See splunkd.log/python.log for more details."}]}
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/mscs_azure_audit.py", line 21, in <module>
schema_para_list=("description",),
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_mod_input.py", line 232, in main
log_suffix=log_suffix,
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_mod_input.py", line 130, in run
tconfig = tc.create_ta_config(settings, config_cls or tc.TaConfig, log_suffix)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 228, in create_ta_config
return config_cls(meta_config, settings, stanza_name, log_suffix)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 53, in __init__
self._load_task_configs()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_config.py", line 75, in _load_task_configs
config_handler = th.ConfigSchemaHandler(self._meta_config, self._client_schema)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_helper.py", line 95, in __init__
self._load_conf_contents()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/data_collection/ta_helper.py", line 120, in _load_conf_contents
self._all_conf_contents = self._config.load()
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/config.py", line 143, in load
log(msg, level=logging.ERROR, need_tb=True)
File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/lib/splunktaucclib/config.py", line 64, in log
stack = "".join(traceback.format_stack())
NoneType: None
~~~

Supplementary Information

The results of `curl` commands on the Heavy Forwarder server are as follows:

• curl -k https://<IP address of the Search Head server>:8089/services/server/info
  • → 200
• curl -k https://<IP address of the Indexer server>:8089/services/server/info
  • → 200
• curl -k https://127.0.0.1:8089/services/server/info
  • → 401 (Unauthorized)
• curl -k https://<IP address of the Search Head server>/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs_settings?count=-1
  • 401 (Unauthorized)
• curl -k https://<IP address of the Indexer server>:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs_settings?count=-1
  • → 401 (Unauthorized)
• curl -k https://127.0.0.1:8089/servicesNS/nobody/Splunk_TA_microsoft-cloudservices/splunk_ta_mscs_settings?c...
  • → 401 (Unauthorized)

If you need any further adjustments or specific aspects to focus on, please let me know!

[tej57]

Hello @km,

I don't think there's any need for resolving the #Concern 1 using web.conf and point the management port of the search head. Since the TA is not functioning as of now, I would suggest to uninstall the TA from HF and directly hit the server/info endpoint on the HF itself. Does that result into 200? If not, there's your problem and there can be different reasons for not getting successful connection. Maybe your splunkd process is terminated and in dangling situation or maybe different other reasons. 

Please check the local connection first after reverting the web.conf change and let us know the output and we can troubleshoot further.

Thanks,
Tejas.