TA_Azure_Monitor - script running failed (exited with code 1).

[morethanyell]

Hi everyone,

An HF node in our env started getting this message all of a sudden.

Unable to initialize modular input "azure_monitor_metrics" defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_monitor_metrics: script running failed (exited with code 1).

While I understand that the articles below seem to have ended with a solution, I feel like I didn't get any procedure on how to attack this problem and solve it.

These are some of the artifacts I found in our internal logs.

09-25-2019 11:59:33.957 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_monitor_metrics"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_monitor_metrics: script running failed (exited with code 1).
09-25-2019 11:59:33.856 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_diagnostic_logs"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_diagnostic_logs: Unable to run `"/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_diagnostic_logs.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.856 +1000 ERROR ModularInputs - Introspecting scheme=azure_diagnostic_logs: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_diagnostic_logs.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.854 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_activity_log"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_activity_log: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_activity_log.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.854 +1000 ERROR ModularInputs - Introspecting scheme=azure_activity_log: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_activity_log.sh --scheme": child failed to start: Permission denied

The box affected is a RHEL.

Thanks in advance.

Article 1. https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/73
Article 2. https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/63

[ivanreis]

Per the log information you had posted, there is a denied permission to run the child process "child failed to start: Permission denied". This lack of permission can be related with "exit code 1" from the app it self.
Verify if the user that is running splunk service has the proper permission to run this TA app, it is common problem in linux environment for permission issues, but I dont know how the permission works at Azure because I never play with before. I would stop splunk service, re-apply the permissions and start splunk service.

[jacobpevans]

Not a full answer, but try setting ModularInputs component logging to DEBUG and see if you get any more useful information. You can do this through the GUI (Server Settings > Logging)

[morethanyell]

Will do. Thanks.