topic Re: TA-ms-loganalytics stops sending data in Splunk Dev

Why does TA-ms-loganalytics stops sending data?

jaihingorani — Tue, 19 Apr 2022 15:48:13 GMT

hope someone can help here.

ta-ms-loganalytics have suddenly stopped working, i can see below type of errors being logged about the modular inputs

ERROR ModularInputs - Unable to initialize modular input "log_analytics" defined inside the app "TA-ms-loganalytics": Introspecting scheme=log_analytics: script running failed (killed by signal 9: Killed).

raise ConnectionError(e, request=request)\nConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=8089): Max retries exceeded with url: /servicesNS/nobody/TA-ms-loganalytics/data/inputs/log_analytics?count=0&output_mode=json (Caused by NewConnectionError('<solnlib.packages.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f118318e610>: Failed to establish a new connection: [Errno 111] Connection refused',))\n"

any help much appreciated.

Re: TA-ms-loganalytics stops sending data

jkat54 — Fri, 08 Apr 2022 15:22:31 GMT

It says it's not able to establish a connection. Did firewall rules change? Can you curl the azure api from the host?

Re: TA-ms-loganalytics stops sending data

jaihingorani — Fri, 08 Apr 2022 15:25:27 GMT

Thank You @jkat54 for responding, we checked with our network teams, there were no changes related to firewalls, we have restarted the splunk services, and we dont see these connection errors anymore, its now logging "ERROR ModularInputs - Unable to initialize modular input "log_analytics" defined inside the app "TA-ms-loganalytics": Introspecting scheme=log_analytics: script running failed (killed by signal 9: Killed)."

Re: TA-ms-loganalytics stops sending data

jaihingorani — Fri, 08 Apr 2022 15:28:33 GMT

curl the azure API -> can you give an example how can i check this ?

Re: TA-ms-loganalytics stops sending data

jaihingorani — Tue, 12 Apr 2022 12:54:00 GMT

Hello again @jkat54

it turned out to be some issues at the OS end, as the CLI was responding slow as well, the logs started to come in post the server reboot. But few of the sources are having the errors as below:

2022-04-12 13:22:23,902 ERROR pid=26000 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="My_Diagnostics" status="400" step="Post Query" response="{"error":{"message":"Response size too large","code":"ResponseSizeError","correlationId":"<myID>","innererror":{"code":"ResponseSizeError","message":"Maximum response size of 100000000 bytes exceeded. Actual response Size is 107148567 bytes."}}}"

Can you suggest if this is something which we can tweak the values for ?

Re: TA-ms-loganalytics stops sending data

jkat54 — Tue, 12 Apr 2022 13:50:01 GMT

you can reduce the amount of data the query returns by adjusting the query.

Re: TA-ms-loganalytics stops sending data

jkat54 — Tue, 12 Apr 2022 13:51:16 GMT

https://docs.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get

Re: TA-ms-loganalytics stops sending data

jaihingorani — Tue, 19 Apr 2022 14:36:04 GMT

Thank you @jkat54 , we have been considering this option, but turns out that our internal teams requires all the data which was being pulled. Is there any way we can increase this limit instead ?

Re: TA-ms-loganalytics stops sending data

jkat54 — Tue, 19 Apr 2022 15:07:31 GMT

I'm not super familiar with log analytics queries.

can you limit the time frame of the query?

in splunk you have earliest, latest, etc. can you grab half the data with one query and the other half with another?

Re: TA-ms-loganalytics stops sending data

jaihingorani — Thu, 21 Apr 2022 11:09:50 GMT

but the inputs are running at every 10minutes of interval, setting the timestamp in the query will not work, because if we do so it will pull the same timeframe of data every 10minutes. is there any way to increase the response size in the TA-ms-loganalytics add on ?

Re: TA-ms-loganalytics stops sending data

jkat54 — Thu, 21 Apr 2022 11:50:10 GMT

Does the query pull all the data every 10 minutes? Instead of getting data for last 10m?

If you run it every 5 minutes will it help?

Please note the size limitation is not from my app, it's the limit on the azure API. You have to reduce the results your query returns or it will always have this error.

you should appeal to azure support if can't figure out how to reduce the size of your query, and can't increase the limit on the API.

There's literally nothing I can do on my end to fix this issue.

Re: TA-ms-loganalytics stops sending data

jaihingorani — Thu, 21 Apr 2022 12:04:24 GMT

Thanks @jkat54 , and understood its the API limit.

the interval is changed to 5 minutes, and its still the same. should the value of event_delay_lag_time = 15 also need to be reduced ?

Re: TA-ms-loganalytics stops sending data

jkat54 — Thu, 21 Apr 2022 12:09:39 GMT

Hear me out...

if you run a query that generates more events than the limit, it will not matter how often or even when you run that query.

so what you have to do, is change your query or increase the limit.

Re: TA-ms-loganalytics stops sending data

jaihingorani — Thu, 21 Apr 2022 12:15:29 GMT

Thanks again, we will explore the possibilities to change the query.