topic Re: Errors after Microsoft Azure Add on for Splunk upgrade in All Apps and Add-ons

Errors after Microsoft Azure Add on for Splunk upgrade

andreeaf1301 — Thu, 11 Nov 2021 13:35:06 GMT

Hello,

We upgraded Microsoft Azure Add on for Splunk to the latest version 3.2.0

After the upgrade, we started seeing the following errors:

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: "Failed to get password of realm=%s, user=%s." % (self._realm, user)

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/aob_py3/solnlib/utils.py", line 148, in wrapper

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: .

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: WARNING:root:Run function: get_password failed: Traceback (most recent call last):

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: solnlib.credentials.CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#TA-MS-AAD#configs/conf-ta_ms_aad_settings, user=proxy.

I tried to add again the credentials and re-create the inputs, but still getting them.

We are getting the logs, but I'm not sure if this errors is impacting us/if we are getting all the logs or how should we correct it.

Thank you,

Andreea

Re: Errors after Microsoft Azure Add on for Splunk upgrade

sloshburch — Tue, 16 Nov 2021 11:35:50 GMT

Curious - is this in Splunk Cloud or is this add-on installed to your own installation of Splunk?

Re: Errors after Microsoft Azure Add on for Splunk upgrade

andreeaf1301 — Tue, 16 Nov 2021 11:54:43 GMT

In Splunk Cloud.

Is it possible to be a new feature of the add-on?

Re: Errors after Microsoft Azure Add on for Splunk upgrade

sloshburch — Tue, 16 Nov 2021 12:58:55 GMT

I don't know about the features of the app/add-on. I've found the developer and asked them to peek at this conversation to help.

Meanwhile, if it's Cloud you may want to add more details to help the developer understand what's going on. Is it Classic or Victoria? This page will help answer that: https://docs.splunk.com/Documentation/SplunkCloud/latest/Admin/Experience

How frequently do you see these error messages occurring? What search did you run to see them?

It's possible that the app upgrade is not related and rather this may be something that has been occurring for a while but only noticed recently. I say this because you highlighted that the data is still working and you only recently noticed these errors.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

andreeaf1301 — Tue, 16 Nov 2021 13:21:19 GMT

- It is classic;

- We are usually classifying the errors we have in our environment with priorities from high to noise. A report is ran and reviewed everyday and everything that was never seen before, appears as new. When we had the upgrade of the add-on to the latest version, I started seeing these errors.

I started investigating them by using the following searches and the date when the errors started to appear matches the timestamp of the upgrade.

index=_internal sourcetype=splunkd "splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py" log_level=error component=PersistentScript

index=_internal sourcetype=splunkd ""Failed to get password of realm=%s, user=%s." % (self._realm, user)" component=PersistentScript log_level=error

Right after the upgrade, the data collection was stopped.

I added again the credentials and the feeds were back to normal, but the errors are still coming.

I'm concerned with them because I don't know if there is any impact in the data we are collecting and I wasn't able to find anything in the documentation.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

sloshburch — Tue, 16 Nov 2021 15:34:29 GMT

One more question - did you notice which host the errors are coming from? I ask because it's possible the add-on is on the search head for it's knowledge objects but the inputs.conf are all stripped away, which might be causing some exceptions.

In Classic, data collection add-ons are installed and managed on the Inputs Data Manager (IDM) so I assume everything on that host is working ok?

Re: Errors after Microsoft Azure Add on for Splunk upgrade

andreeaf1301 — Tue, 16 Nov 2021 15:42:15 GMT

We have the add-on installed and configured on IDM. That's where the errors are coming from.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

jconger — Tue, 16 Nov 2021 23:23:54 GMT

The error messages are erroneous.

TL;DR = to work around this issue, add the following in local/ta_ms_aad_settings.conf:

[proxy]
proxy_password = None

This error is caused by a library used by add-ons built with add-on builder (a.k.a. AoB) that implement a proxy. In a nutshell, the REST handler settings code generated by AoB (in this case TA_MS_AAD_rh_settings.py) defines the proxy fields. Here is what it looks like for proxy_password:

field.RestField( 'proxy_password', required=False, encrypted=True, default=None, validator=validator.String( min_len=0, max_len=8192, ) )

The proxy_password settings specify that the field is not required, but the field is encrypted. Now, getting back to the library code. The library is trying to decrypt proxy settings even if you haven't specified any. An internal bug has been filed, but the workaround should stop the (erroneous) error messages.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

carlkennedy_con — Fri, 18 Feb 2022 14:51:19 GMT

This solution worked for me.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

jabezds — Thu, 06 Oct 2022 09:54:16 GMT

@jconger Thanks for the solution,

Do you think reconfiguring the proxy settings on the Addon after the upgrade will also solve this issues?

Re: Errors after Microsoft Azure Add on for Splunk upgrade

TLepingwell — Mon, 27 Feb 2023 20:12:11 GMT

For general interest. This bug is still present in version 1.2.5 of the app. The filenames have changed slightly, the file to which the stanza must be added is now called ta_microsoft_graph_security_add_on_for_splunk_settings.conf

Note that this file is only created by the app if you add a proxy setting, otherwise you will have to manually create it. It is different from the similarly named ta_microsoft_graph_security_add_on_for_splunk_account.conf configuration file.

Re: Errors after Microsoft Azure Add on for Splunk upgrade

Cerum — Thu, 27 Feb 2025 14:59:00 GMT

I'm having a similar issue with the Egnyte Collaborate TA - https://splunkbase.splunk.com/app/5653.

When trying to use the add-on - I keep getting the following error:

“01-22-2025 16:45:55.409 +0000 ERROR PersistentScript [1693337 PersistentScriptIo] - From {/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/TA-egnyte-connect/bin/TA_egnyte_connect_rh_settings.py persistent}: solnlib.credentials.CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#TA-egnyte-connect#configs/conf-ta_egnyte_connect_settings, user=proxy.”

The issue is I'm running Splunk Cloud - and don't have the ability to modify local conf files. Any ideas on how to get this resolved for Slunk Cloud customers?

Cheers.