All Apps and Add-ons

Errors enabling Add-ons: Add-on conflicting in Heavy Forwarder TA-ms-loganalytics and Splunk_TA_microsoft-cloudservices.

jaihingorani
Explorer

Hi,

I have two add-ons installed on the same heavy forwarder, Splunk_TA_microsoft-cloudservices v4.0.1 AND TA-ms-loganalytics v1.0.3. I am seeing a conflict when both the add-ons are enabled at the same time. I have cloud services add on installed and running, and when i install the loganalytics add on, the loganalytics doesn't work and cloud services also gets stopped. I am getting the below errors when I enable both the add-ons at the same time.

The errors occur when I enable both of the add-ons at the same time.

Kindly help.

on log analytics addon -

Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 113, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 348, in _format_all_response\n self._encrypt_raw_credentials(cont['entry'])\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/handler.py", line 382, in _encrypt_raw_credentials\n change_list = rest_credentials.decrypt_all(data)\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/rest_handler/credentials.py", line 286, in decrypt_all\n all_passwords = credential_manager._get_all_passwords()\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/utils.py", line 154, in wrapper\n return func(*args, **kwargs)\n File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/solnlib/credentials.py", line 272, in _get_all_passwords\n clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n"

message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/splunktaucclib/global_config/configuration.py", line 270, in load

message from "python /opt/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ucc_inputs = global_config.inputs.load(input_type=self.input_type)

on cloud services add on-

Unexpected error "<class 'splunktaucclib.rest_handler.error.RestError'>" from python handler: "REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 116, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 178, in all\n **query\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 289, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 71, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 679, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1183, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1241, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/splunk_rest_client.py", line 145, in request\n verify=verify, proxies=proxies, cert=cert, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/api.py", line 60, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine("''",))\n". See splunkd.log for more details.

Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 93, in init_persistent\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 594, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunk_ta_mscs_rh_mscs_storage_blob.py", line 114, in handleList\n AdminExternalHandler.handleList(self, confInfo)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/admin_external.py", line 51, in wrapper\n for entity in result:\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 123, in wrapper\n raise RestError(500, traceback.format_exc())\nRestError: REST Error [500]: Internal Server Error -- Traceback (most recent call last):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 116, in wrapper\n for name, data, acl in meth(self, *args, **kwargs):\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/splunktaucclib/rest_handler/handler.py", line 178, in all\n **query\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 289, in wrapper\n return request_fun(self, *args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 71, in new_f\n val = f(*args, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 679, in get\n response = self.http.get(path, all_headers, **query)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1183, in get\n return self.request(url, { 'method': "GET", 'headers': headers })\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/splunklib/binding.py", line 1241, in request\n response = self.handler(url, message, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/splunk_rest_client.py", line 145, in request\n verify=verify, proxies=proxies, cert=cert, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/api.py", line 60, in request\n return session.request(method=method, url=url, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 533, in request\n resp = self.send(prep, **send_kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/sessions.py", line 646, in send\n r = adapter.send(request, **kwargs)\n File "/opt/splunk/etc/apps/Splunk_TA_microsoft-cloudservices/bin/splunktamscs/solnlib/packages/requests/adapters.py", line 498, in send\n raise ConnectionError(err, request=request)\nConnectionError: ('Connection aborted.', BadStatusLine("''",))\n\n
Labels (3)
0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n"

you have the exception above in the log analytics app.   It's saying that a password being passed is NoneType which means it's failing to retrieve the password from splunk password store, or you haven't provided one.

View solution in original post

jaihingorani
Explorer
0 Karma

jhantuSplunk1
Path Finder

@jkat54 

 

We are also facing same issue unable to get data by this add-on having http 500 Internal server error. Please assist is asap.

0 Karma

jkat54
SplunkTrust
SplunkTrust

clear_password += field_clear[index]\nTypeError: cannot concatenate 'str' and 'NoneType' objects\n"

you have the exception above in the log analytics app.   It's saying that a password being passed is NoneType which means it's failing to retrieve the password from splunk password store, or you haven't provided one.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Both addons use kvstore.  It's possible there is a conflict because you've named the inputs the same and both are trying to use the same kvstore name.

jkat54
SplunkTrust
SplunkTrust

If kvstore is broken in any other way, that can be the issue too.

jhantuSplunk1
Path Finder

Thanks Mate,

We have in a single HFR two Splunk heavy forwarder location

1. /opt/splunk/bin

Here this add-on is working fine

2. /opt/splunk_temp/splunk

But same configuration it is not working here with same KV store configuration & password.conf file

 

 

 

 

 

0 Karma

jhantuSplunk1
Path Finder

Also both side 

./splunk show kvstore-status
Your session is invalid. Please login.
Splunk username:
Password:

This member:
jvjhvhjvjvjvjvjhvhj : Ready
date : Tue Jul 7 00:27:49 2020
dateSec : 1594074469.921
disabled : 0
guid : mmmmm-kkkk-hhhh-jjjjj-fffffffffffff
oplogEndTimestamp : Tue Jul 7 00:27:41 2020
oplogEndTimestampSec : 1594074461
oplogStartTimestamp : Wed Jul 1 12:42:33 2020
oplogStartTimestampSec : 1593600153
port : 8192
replicaSet : vcvccvc-hgcgcjc-vncvcjgc-cgcjcgcc
replicationStatus : KV store captain
standalone : 1
status : ready

KV store members:
xxx.x.x.x:yyyy
configVersion : 1
electionDate : Tue Jul 7 00:01:39 2020
electionDateSec : 1594072899
hostAndPort : xxx.x.x.x:yyyy
optimeDate : Tue Jul 7 00:27:41 2020
optimeDateSec : 1594074461
replicationStatus : KV store captain
uptime : 1570

0 Karma

jhantuSplunk1
Path Finder

Same conf. is working /opt/splunk/bin but why not under /opt/splunk_temp/splunk/bin location's HFR secondary instance now? 

0 Karma

jkat54
SplunkTrust
SplunkTrust

Did you change the ports for kvstore, etc on the 2nd install?

jhantuSplunk1
Path Finder

Yes Yes, i had taken new port for kv store. so do i need to revert to like first instance?

0 Karma

jkat54
SplunkTrust
SplunkTrust

You need to run your kvstore on different ports.

 

if both have 8192 as the port, then only one will be able to bind to the port.  Then the one that can't bind to the port can't use the kvstore because the other kvstore port held hostage.

jhantuSplunk1
Path Finder

First instance - 8191 & second instance - 8192 both are different port and getting this http 500 & this credential issue.

Do you suggest then next step?

 

0 Karma

jkat54
SplunkTrust
SplunkTrust

Restart splunk On cli and read the screen... see if it's mad about a port not being available 

jhantuSplunk1
Path Finder

Thanks Mate,

Checking kvstore port [8192]: open

Restart time i am able to get this message. the same password.conf file whichone is working in my first instance same one i am using second instance hence trying to fixing this issue.

 

0 Karma

jhantuSplunk1
Path Finder

Hi,

After fixing all errors then getting below 

 


2020-07-07 03:06:25.203 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORGet Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: b6e89b21-ad56-42fe-a006-3177419a8600\r\nCorrelation ID: 68a7a5b7-4acb-4fdc-abf7-5a77edce87b3\r\nTimestamp: 2020-07-07 01:06:25Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:25Z","trace_id":"b6e89b21-ad56-42fe-a006-3177419a8600","correlation_id":"68a7a5b7-4acb-4fdc-abf7-5a77edce87b3","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: b6e89b21-ad56-42fe-a006-3177419a8600\r\nCorrelation ID: 68a7a5b7-4acb-4fdc-abf7-5a77edce87b3\r\nTimestamp: 2020-07-07 01:06:25Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:25Z","trace_id":"b6e89b21-ad56-42fe-a006-3177419a8600","correlation_id":"68a7a5b7-4acb-4fdc-abf7-5a77edce87b3","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" raise AdalError(return_error_string, error_response) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/oauth2_client.py", line 281, in get_token HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return client.get_token(oauth_parameters) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/token_request.py", line 113, in _oauth_get_token HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" token = self._oauth_get_token(oauth_parameters) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/token_request.py", line 316, in get_token_with_client_credentials HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return token_request.get_token_with_client_credentials(client_secret) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 158, in token_func HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return token_func(self) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 109, in _acquire_token HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" return self._acquire_token(token_func) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/adal/authentication_context.py", line 160, in acquire_token_with_client_credentials HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 49, in collect_events HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" input_module.collect_events(self, ew) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" self.collect_events(ew) HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" File "/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events HFR01
2020-07-07 03:06:25.178 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" Traceback (most recent call last): HFR01
2020-07-07 03:06:23.846 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" ERRORGet Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: bf76db00-dd5d-46fa-a8e7-789d40cf7c00\r\nCorrelation ID: f707f17a-874b-420c-943f-0131e62a565c\r\nTimestamp: 2020-07-07 01:06:23Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:23Z","trace_id":"bf76db00-dd5d-46fa-a8e7-789d40cf7c00","correlation_id":"f707f17a-874b-420c-943f-0131e62a565c","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01
2020-07-07 03:06:23.821 message from "python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py" AdalError: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided.\r\nTrace ID: bf76db00-dd5d-46fa-a8e7-789d40cf7c00\r\nCorrelation ID: f707f17a-874b-420c-943f-0131e62a565c\r\nTimestamp: 2020-07-07 01:06:23Z","error_codes":[7000215],"timestamp":"2020-07-07 01:06:23Z","trace_id":"bf76db00-dd5d-46fa-a8e7-789d40cf7c00","correlation_id":"f707f17a-874b-420c-943f-0131e62a565c","error_uri":"https://login.microsoftonline.com/error?code=7000215"} HFR01
2020-07-07 03:06:20.190 New scheduled exec process: python /opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/bin/log_analytics.py HFR01
2020-07-07 03:06:19.513 Found external scheme definition for stanza="log_analytics://" from spec file="/opt/splunk_temp/splunk/etc/apps/TA-ms-loganalytics/README/inputs.conf.spec" with parameters="resource_group, workspace_id, subscription_id, tenant_id, application_id, application_key, log_analytics_query, start_date, event_delay_lag_time"

 

0 Karma

jkat54
SplunkTrust
SplunkTrust

Great, now the issue is still just a matter of googling the error in your logs:

error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret is provided"

jhantuSplunk1
Path Finder

This has been successfully Fixed.

  1. Kv store port issue - Fixed
  2. Password issue -Fixed
  3. Datetime /start time issue -Fixed
  4. error_description":"AADSTS7000215: Invalid client secret is provided – Fixed
  5. Secret key issue has been fixed
  6. Password issue has been fixed

jkat54
SplunkTrust
SplunkTrust

Perfect!

I accepted my first answer as the answer since the original issue was password / kvstore related.

Also thumbs upping your final solution list because its a good list for anyone else troubleshooting this.

jhantuSplunk1
Path Finder

Excellent discussion and your appreciated assistance @jkat54. Could you share me url more details configuration on inputs.conf (startdate) and checkpoint file.

I wanted to try recovery some backlogs data by using this second instance now both are working. But i wanted second instance LA should configure such way so that i could recover my backlogs data example i wanted recovery my april 2020 15th  april to 20th april data from LA Workspace from Azure side.

While i am validating data its first pulling current data and then go-ahead to some backward but not pulling all backlogs data.

Can i customizing some backlogs time duration pulling data?

 

 

0 Karma

jkat54
SplunkTrust
SplunkTrust

See inputs.conf.spec under the README folder.

See this link for working with kvstore:

https://docs.splunk.com/Documentation/Splunk/8.0.4/RESTREF/RESTkvstore

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!