Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

[phularah]

Are there any specific ports or specific permissions this add-on requires/uses, so that I can inform the team, so if any modifications are made data flow is not interrupted.

I have configured Microsoft Log Analytics Add-on in Heavy Forwarder and forwarding the logs received to indexer. There is no clustering. I would like to hear from @jkat54 and @dpanych. Any ideas, why this keep on happening.

I used

index=_internal log_level=err* OR log_level=warn loganalytics*

The latest event I am getting some results using this query is

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[phularah]

Again, data has stopped coming and using index=_internal log_level=err* loganalytics* gives these errors: Today date is 9/13/2018. and the last data is of 9/12/18
4:18:13.990 PM.

09-12-2018 08:38:10.336 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERRORlocal variable 'data' referenced before assignment

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" UnboundLocalError: local variable 'data' referenced before assignment

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" for i in range(len(data["tables"][0]["rows"])):

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 86, in collect_events

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" input_module.collect_events(self, ew)

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" self.collect_events(ew)

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events

09-12-2018 08:38:09.834 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" Traceback (most recent call last):

09-12-2018 06:41:01.718 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERRORlocal variable 'data' referenced before assignment

[phularah]

Sometimes, it breaks in 2-4 days, sometimes in 15-16 hours.

[phularah]

Also, for source=splunkd, we are getting these messages

09-10-2018 08:02:41.053 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERRORGet Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS70001: Application with identifier '37e37c43-5946-483a-a856-041490e76e8cccc' was not found in the directory 30f52344-4663-4c2e-bab3-61bf24ebbed8\r\nTrace ID: 4327f55a-bb53-4606-b506-66fc1b4e0500\r\nCorrelation ID: 6ec81c9d-4f8a-47ea-84b4-2ad2b7e40a3e\r\nTimestamp: 2018-09-10 06:02:40Z","error_codes":[70001],"timestamp":"2018-09-10 06:02:40Z","trace_id":"4327f55a-bb53-4606-b506-66fc1b4e0500","correlation_id":"6ec81c9d-4f8a-47ea-84b4-2ad2b7e40a3e"}

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" raise AdalError(return_error_string, error_response)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\oauth2_client.py", line 281, in get_token

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" return client.get_token(oauth_parameters)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 113, in _oauth_get_token

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" token = self._oauth_get_token(oauth_parameters)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 316, in get_token_with_client_credentials

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" return token_request.get_token_with_client_credentials(client_secret)

[phularah]

no...again we faced an issue..data again stopped coming even though we haven't changed anything..
We are receiving below errors from sourcetype="ta:ms:loganalytics:log"

2018-09-10 08:01:40,148 ERROR pid=11372 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events
input_module.collect_events(self, ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 49, in collect_events
token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 160, in acquire_token_with_client_credentials
return self._acquire_token(token_func)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 109, in _acquire_token
return token_func(self)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 158, in token_func
return token_request.get_token_with_client_credentials(client_secret)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 316, in get_token_with_client_credentials
token = self._oauth_get_token(oauth_parameters)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 113, in _oauth_get_token
return client.get_token(oauth_parameters)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\oauth2_client.py", line 281, in get_token
raise AdalError(return_error_string, error_response)
AdalError: Get Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS70001: Application with identifier '37e37c43-5946-483a-a856-041490e76e8cccc' was not found in the directory 30f52344-4663-4c2e-bab3-61bf24ebbed8\r\nTrace ID: 3cdc5a4c-98df-4102-916f-779ce15e0500\r\nCorrelation ID: 403f848a-a918-4d61-8a85-164c1df79e29\r\nTimestamp: 2018-09-10 06:01:40Z","error_codes":[70001],"timestamp":"2018-09-10 06:01:40Z","trace_id":"3cdc5a4c-98df-4102-916f-779ce15e0500","correlation_id":"403f848a-a918-4d61-8a85-164c1df79e29"}