topic Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS? in All Apps and Add-ons

Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Thu, 06 Sep 2018 06:56:26 GMT

Are there any specific ports or specific permissions this add-on requires/uses, so that I can inform the team, so if any modifications are made data flow is not interrupted.

I have configured Microsoft Log Analytics Add-on in Heavy Forwarder and forwarding the logs received to indexer. There is no clustering. I would like to hear from @jkat54 and @dpanych. Any ideas, why this keep on happening.

I used

index=_internal log_level=err* OR log_level=warn loganalytics*

The latest event I am getting some results using this query is

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Thu, 06 Sep 2018 07:56:39 GMT

It connects to the log analytics API on TCP port 443 aka HTTPS.

Nothing else is needed.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Thu, 06 Sep 2018 08:15:02 GMT

I am not sure why is it happening then. I tried making a new input but still I am unable to see any data. Any idea why it might be happening?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Tue, 29 Sep 2020 21:11:53 GMT

After making new inputs and deleting old inputs, I am getting data now. But, I don't know why it stopped in the first place and now after first making a new input and disabling previous input, data didn't come. I again made a new input after deleting all disabled inputs, now I am getting data. It is really frustrating, and I am unable to pinpoint the source.

Now, new results are coming where log_level=warn,

09-06-2018 10:50:34.492 +0200 WARN LineBreakingProcessor - Truncating line because limit of 10000 bytes has been exceeded with a line length >= 14261 - data_source="log_analytics://analytics", data_host="Hostname", data_sourcetype="loganalytics"

but it is of no concern, my concern is this error.
Could you please tell why this error might show up.

09-05-2018 18:24:24.168 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERROR('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Thu, 06 Sep 2018 10:33:05 GMT

Sounds like the checkpoints were messed up. Did you upgrade from a previous version of the app or install fresh/new?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Thu, 06 Sep 2018 10:36:03 GMT

Yes, I had upgraded the add-on. I was using the latest add-on before this problem surfaced.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Thu, 06 Sep 2018 14:58:32 GMT

Ok looks like the process for upgrading should have been to delete the inputs, upgrade the app, add the inputs.

this is due to the way the checkpoints changed between versions. My apologies for the inconvenience.

Is every input you’ve re-added working now?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Sat, 08 Sep 2018 11:32:29 GMT

Was everything working after removing and adding the inputs after the upgrade?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Tue, 29 Sep 2020 21:12:48 GMT

no...again we faced an issue..data again stopped coming even though we haven't changed anything..
We are receiving below errors from sourcetype="ta:ms:loganalytics:log"

2018-09-10 08:01:40,148 ERROR pid=11372 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events
input_module.collect_events(self, ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 49, in collect_events
token_response = context.acquire_token_with_client_credentials('https://api.loganalytics.io/', application_id, application_key)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 160, in acquire_token_with_client_credentials
return self._acquire_token(token_func)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 109, in _acquire_token
return token_func(self)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\authentication_context.py", line 158, in token_func
return token_request.get_token_with_client_credentials(client_secret)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 316, in get_token_with_client_credentials
token = self._oauth_get_token(oauth_parameters)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\token_request.py", line 113, in _oauth_get_token
return client.get_token(oauth_parameters)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\oauth2_client.py", line 281, in get_token
raise AdalError(return_error_string, error_response)
AdalError: Get Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS70001: Application with identifier '37e37c43-5946-483a-a856-041490e76e8cccc' was not found in the directory 30f52344-4663-4c2e-bab3-61bf24ebbed8\r\nTrace ID: 3cdc5a4c-98df-4102-916f-779ce15e0500\r\nCorrelation ID: 403f848a-a918-4d61-8a85-164c1df79e29\r\nTimestamp: 2018-09-10 06:01:40Z","error_codes":[70001],"timestamp":"2018-09-10 06:01:40Z","trace_id":"3cdc5a4c-98df-4102-916f-779ce15e0500","correlation_id":"403f848a-a918-4d61-8a85-164c1df79e29"}

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Tue, 29 Sep 2020 21:12:53 GMT

Also, for source=splunkd, we are getting these messages

09-10-2018 08:02:41.053 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" ERRORGet Token request returned http error: 400 and server response: {"error":"unauthorized_client","error_description":"AADSTS70001: Application with identifier '37e37c43-5946-483a-a856-041490e76e8cccc' was not found in the directory 30f52344-4663-4c2e-bab3-61bf24ebbed8\r\nTrace ID: 4327f55a-bb53-4606-b506-66fc1b4e0500\r\nCorrelation ID: 6ec81c9d-4f8a-47ea-84b4-2ad2b7e40a3e\r\nTimestamp: 2018-09-10 06:02:40Z","error_codes":[70001],"timestamp":"2018-09-10 06:02:40Z","trace_id":"4327f55a-bb53-4606-b506-66fc1b4e0500","correlation_id":"6ec81c9d-4f8a-47ea-84b4-2ad2b7e40a3e"}

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" raise AdalError(return_error_string, error_response)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\adal\oauth2_client.py", line 281, in get_token

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" return client.get_token(oauth_parameters)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" token = self._oauth_get_token(oauth_parameters)

09-10-2018 08:02:40.553 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" return token_request.get_token_with_client_credentials(client_secret)

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

493669 — Mon, 10 Sep 2018 06:59:40 GMT

@jkat54,
is there any retry attempts for input in OMS?
like due to some reason in oms, splunk unable to collect data for 5 min. but after 5 min. everything fine at OMS side...but then splunk unable to receive any kind of data ..it get stopped so is there any retry attempt like it will try to connect with OMS for few attempts and then it will stop attemptting to connect with OMS?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Mon, 10 Sep 2018 10:26:10 GMT

The “connection broken” error suggests a proxy or firewall or other network issue.

No one else is reporting this error so I believe it to be something with your environment only.

As for if the add on retries connections, no it only attempts one connection per interval.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Mon, 10 Sep 2018 11:02:57 GMT

Wait... this question wasn’t by you. @493669 do you work with phularah?

If not, you should create your own question.

If yes, then see this link for how to resolve the error you have: https://docs.microsoft.com/en-us/azure/active-directory/application-sign-in-problem-federated-sso-gallery#application-not-found-in-directory

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Mon, 10 Sep 2018 11:07:11 GMT

Yes, @jkat me and @493669 are working together. My comments were sent to moderator, so I asked my colleague @493669 to post these comments. Now, I can see my posted comments.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Mon, 10 Sep 2018 11:21:32 GMT

@jkat54 , but if we have any error at oms side then data will not come... but after I deleted previous input and created new input , data started flowing again...so it doesn't seems to be issue at OMS side..isn't it?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Mon, 10 Sep 2018 11:45:08 GMT

When you recreate do you recreate in splunk only?

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Mon, 10 Sep 2018 11:49:41 GMT

Yes, we create new inputs in Splunk only.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Tue, 11 Sep 2018 05:37:01 GMT

can you please share your take on this, why it might be happening? I had a chat with Azure guys in my team, they say everything is working fine at their side and they are not making any changes.

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

phularah — Tue, 29 Sep 2020 21:10:38 GMT

Also, we are getting these messages running this query, index=_internal log_level=err* loganalytics*. Again after working for a few hours data has again stopped coming.

These are the error messages, we are getting:

9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py" UnboundLocalError: local variable 'data' referenced before assignment

    host =  *****   
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     for i in range(len(data["tables"][0]["rows"])):

    host =  ********    
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 86, in collect_events

    host =  ******  
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     input_module.collect_events(self, ew)

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events

    host =  *****
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd 

    9/11/18
5:59:55.927 AM  
09-11-2018 05:59:55.927 +0200 ERROR ExecProcessor - message from "python F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py"     self.collect_events(ew)

    host =  *******
    source =    F:\Splunk\var\log\splunk\splunkd.log    
    sourcetype =    splunkd

Re: Microsoft Log Analytics Ad-on: Why does the data stop coming in Splunk after firewall rules are modified in OMS?

jkat54 — Tue, 11 Sep 2018 08:34:27 GMT

This means the query didn’t return any results.

Any idea how long it takes before it breaks?