Errors enabling Add-ons: Add-on conflicting in Heavy Forwarder TA-ms-loganalytics and Splunk_TA_microsoft-cloudservices.

[jaihingorani]

Hi,

I have two add-ons installed on the same heavy forwarder, Splunk_TA_microsoft-cloudservices v4.0.1 AND TA-ms-loganalytics v1.0.3. I am seeing a conflict when both the add-ons are enabled at the same time. I have cloud services add on installed and running, and when i install the loganalytics add on, the loganalytics doesn't work and cloud services also gets stopped. I am getting the below errors when I enable both the add-ons at the same time.

The errors occur when I enable both of the add-ons at the same time.

Kindly help.

on log analytics addon -

Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 348, in _format_all_response\n self._encrypt_raw_credentials(cont['entry'])\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 382, in _encrypt_raw_credentials\n change_list = rest_credentials.decrypt_all(data)\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/credentials.py", line 286, in decrypt_all\n all_passwords = credential_manager._get_all_passwords()\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n"

message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 270, in load

message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ucc_inputs = global_config.inputs.load(input_type=self.input_type)

on cloud services add on-

Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 116, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 178, in all\n **query\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 289, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 71, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 679, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1183, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1241, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/splunk_rest_client.py", line 145, in request\n verify=verify, proxies=proxies, cert=cert, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/api.py", line 60, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine("''",))\n". See splunkd.log for more details.

Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 93, in init_persistent\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 594, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunk_ta_mscs_rh_mscs_storage_blob.py", line 114, in handleList\n AdminExternalHandler.handleList(self, confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/admin_external.py", line 51, in wrapper\n for entity in result:\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 123, in wrapper\n raise RestError(500, traceback.format_exc())\nRestError: REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 116, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 178, in all\n **query\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 289, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 71, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 679, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1183, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1241, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/splunk_rest_client.py", line 145, in request\n verify=verify, proxies=proxies, cert=cert, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/api.py", line 60, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine("''",))\n\n

[jhantuSplunk1]

Thanks @jkat54 

[jkat54]

You can "thank" me by clicking the thumbs up button on my replies / answers 😉

[jaihingorani]

Thanks, the solution worked!!!

[jhantuSplunk1]

Thanks mate...

[jkat54]

Your problem is with your AAD setup now, and that's not splunk or my app so I can't help you with that.

[jhantuSplunk1]

Actually i have different port for two instance. First instance is having 8191 kvstore  port & this is working. But second one is having 8192 port is havinf error.

2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"<class 'splunktaucclib.rest_handler.error.RestError'>\" from python handler: \"REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 348, in _format_all_response\n self._encrypt_raw_credentials(cont['entry'])\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py\", line 382, in _encrypt_raw_credentials\n change_list = rest_credentials.decrypt_all(data)\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/credentials.py\", line 286, in decrypt_all\n all_passwords = credential_manager._get_all_passwords()\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/utils.py\", line 154, in wrapper\n return func(*args, **kwargs)\n File \"/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/credentials.py\", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n\". See splunkd.log for more details."}]}HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" raise HTTPError(response) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 1221, in request HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return self.request(url, { 'method': "GET", 'headers': headers }) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 1160, in get HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" response = self.http.get(path, self._auth_headers, **query) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 665, in get HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" val = f(*args, **kwargs) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 69, in new_f HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return request_fun(self, *args, **kwargs) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/packages/splunklib/binding.py", line 287, in wrapper HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" **query HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 175, in _load_endpoint HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_item['entity'] HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 270, in load HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ucc_inputs = global_config.inputs.load(input_type=self.input_type) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 171, in _parse_input_args_from_global_config HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self._parse_input_args_from_global_config(inputs) HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 152, in parse_input_args HFR01
2020-07-07 01:10:46 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.parse_input_args(input_definition)

[jkat54]

Still failing to get the password from the kvstore.  Try adding the password again.  Try fixing problems with kvstore. 

the issue is your setup, not the app.  Since you're off the beaten path, running two instances on one machine, you'll need to solve this yourself.  It's not a supported setup. 

[jhantuSplunk1]

For first instance i am getting below error but still able to see data flow.

2020-07-07 01:35:59.377 message from "/opt/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" Modular input azure_diagnostic_logs://Azure Monitor Diagnostic Logs _ GOEP DEV Error getting event hub creds: Error: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r...

[jhantuSplunk1]

My bad, yes seem to be i have taken new port in second instance. So can i update this port to like first working instance?