All Apps and Add-ons

TA_Azure_Monitor - script running failed (exited with code 1).

morethanyell
Contributor

Hi everyone,

An HF node in our env started getting this message all of a sudden.

Unable to initialize modular input "azure_monitor_metrics" defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_monitor_metrics: script running failed (exited with code 1).

While I understand that the articles below seem to have ended with a solution, I feel like I didn't get any procedure on how to attack this problem and solve it.

These are some of the artifacts I found in our internal logs.

09-25-2019 11:59:33.957 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_monitor_metrics"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_monitor_metrics: script running failed (exited with code 1).
09-25-2019 11:59:33.856 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_diagnostic_logs"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_diagnostic_logs: Unable to run `"/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_diagnostic_logs.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.856 +1000 ERROR ModularInputs - Introspecting scheme=azure_diagnostic_logs: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_diagnostic_logs.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.854 +1000 ERROR ModularInputs - Unable to initialize modular input "azure_activity_log"  defined inside the app "TA_Azure_Monitor": Introspecting scheme=azure_activity_log: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_activity_log.sh --scheme": child failed to start: Permission denied
09-25-2019 11:59:33.854 +1000 ERROR ModularInputs - Introspecting scheme=azure_activity_log: Unable to run "/opt/splunk/etc/apps/TA_Azure_Monitor/bin/azure_activity_log.sh --scheme": child failed to start: Permission denied

The box affected is a RHEL.

Thanks in advance.

Article 1. https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/73
Article 2. https://github.com/Microsoft/AzureMonitorAddonForSplunk/issues/63

0 Karma

ivanreis
Builder

Per the log information you had posted, there is a denied permission to run the child process "child failed to start: Permission denied". This lack of permission can be related with "exit code 1" from the app it self.
Verify if the user that is running splunk service has the proper permission to run this TA app, it is common problem in linux environment for permission issues, but I dont know how the permission works at Azure because I never play with before. I would stop splunk service, re-apply the permissions and start splunk service.

0 Karma

jacobpevans
Motivator

Not a full answer, but try setting ModularInputs component logging to DEBUG and see if you get any more useful information. You can do this through the GUI (Server Settings > Logging)

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

morethanyell
Contributor

Will do. Thanks.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!