Errors after Microsoft Azure Add on for Splunk upgrade

[andreeaf1301]

Hello,

We upgraded Microsoft Azure Add on for Splunk to the latest version 3.2.0

After the upgrade, we started seeing the following errors:

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: "Failed to get password of realm=%s, user=%s." % (self._realm, user)

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/aob_py3/solnlib/utils.py", line 148, in wrapper

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: .

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: WARNING:root:Run function: get_password failed: Traceback (most recent call last):

From {/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py persistent}: solnlib.credentials.CredentialNotExistException: Failed to get password of realm=__REST_CREDENTIAL__#TA-MS-AAD#configs/conf-ta_ms_aad_settings, user=proxy.

I tried to add again the credentials and re-create the inputs, but still getting them.

We are getting the logs, but I'm not sure if this errors is impacting us/if we are getting all the logs or how should we correct it.

Thank you,

Andreea

[jconger · Accepted solution]

The error messages are erroneous. 

TL;DR = to work around this issue, add the following in local/ta_ms_aad_settings.conf:

[proxy]
proxy_password = None

This error is caused by a library used by add-ons built with add-on builder (a.k.a. AoB) that implement a proxy.  In a nutshell, the REST handler settings code generated by AoB (in this case TA_MS_AAD_rh_settings.py) defines the proxy fields.  Here is what it looks like for proxy_password:

field.RestField(
    'proxy_password',
    required=False,
    encrypted=True,
    default=None,
    validator=validator.String(
        min_len=0, 
        max_len=8192, 
    )
)

The proxy_password settings specify that the field is not required, but the field is encrypted.  Now, getting back to the library code.  The library is trying to decrypt proxy settings even if you haven't specified any.  An internal bug has been filed, but the workaround should stop the (erroneous) error messages.

[TLepingwell]

For general interest. This bug is still present in version 1.2.5 of the app. The filenames have changed slightly, the file to which the stanza must be added is now called ta_microsoft_graph_security_add_on_for_splunk_settings.conf

Note that this file is only created by the app if you add a proxy setting, otherwise you will have to manually create it. It is different from the similarly named ta_microsoft_graph_security_add_on_for_splunk_account.conf configuration file.

[jabezds]

@jconger Thanks for the solution,

Do you think reconfiguring the proxy settings on the Addon after the upgrade will also solve this issues?

[carlkennedy_con]

This solution worked for me.

[andreeaf1301]

- It is classic;

- We are usually classifying the errors we have in our environment with priorities from high to noise. A report is ran and reviewed everyday and  everything that was never seen before, appears as new. When we had the upgrade of the add-on to the latest version, I started seeing these errors.

I started investigating them by using the following searches and the date when the errors started to appear matches the timestamp of the upgrade.

index=_internal sourcetype=splunkd "splunk/etc/apps/TA-MS-AAD/bin/TA_MS_AAD_rh_settings.py" log_level=error component=PersistentScript

index=_internal sourcetype=splunkd ""Failed to get password of realm=%s, user=%s." % (self._realm, user)" component=PersistentScript log_level=error

Right after the upgrade, the data collection was stopped.

I added again the credentials and the feeds were back to normal, but the errors are still coming. 

I'm concerned with them because I don't know if there is any impact in the data we are collecting and I wasn't able to find anything in the documentation.

[sloshburch]

One more question - did you notice which host the errors are coming from? I ask because it's possible the add-on is on the search head for it's knowledge objects but the inputs.conf are all stripped away, which might be causing some exceptions.

In Classic, data collection add-ons are installed and managed on the Inputs Data Manager (IDM) so I assume everything on that host is working ok?

[andreeaf1301]

We have the add-on installed and configured on IDM. That's where the errors are coming from.

[sloshburch]

Curious - is this in Splunk Cloud or is this add-on installed to your own installation of Splunk?

[andreeaf1301]

In Splunk Cloud.

Is it possible to be a new feature of the add-on?

[sloshburch]

I don't know about the features of the app/add-on. I've found the developer and asked them to peek at this conversation to help.

Meanwhile, if it's Cloud you may want to add more details to help the developer understand what's going on. Is it Classic or Victoria? This page will help answer that: https://docs.splunk.com/Documentation/SplunkCloud/latest/Admin/Experience 

How frequently do you see these error messages occurring? What search did you run to see them?

It's possible that the app upgrade is not related and rather this may be something that has been occurring for a while but only noticed recently. I say this because you highlighted that the data is still working and you only recently noticed these errors.