topic Re: TA-ms-loganalytics data ingestion issue for few sourcetypes in Splunk Enterprise

TA-ms-loganalytics data ingestion issue for few sourcetypes

jaihingorani — Fri, 12 Mar 2021 12:41:41 GMT

i am having issues where some of the sourcetypes are not getting data in splunk from LA, upon checking some logs i can see below :

2021-03-11 13:07:39,577 ERROR pid=13031 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="MyInput" status="400" step="Post Query" response="{"error":{"message":"Response size too large","code":"ResponseSizeError","correlationId":"XXX","innererror":{"code":"ResponseSizeError","message":"Maximum response size of 67108864 bytes exceeded. Actual response Size is 73664723 bytes."}}}"
2021-03-11 13:07:39,577 ERROR pid=13031 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events
input_module.collect_events(self, ew)
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 86, in collect_events
for i in range(len(data["tables"][0]["rows"])):
UnboundLocalError: local variable 'data' referenced before assignment

2021-03-11 13:08:18,608 ERROR pid=13216 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="MyInput2" status="400" step="Post Query" response="{"error":{"message":"Response size too large","code":"ResponseSizeError","correlationId":"XXX","innererror":{"code":"ResponseSizeError","message":"Maximum response size of 67108864 bytes exceeded. Actual response Size is 73136457 bytes."}}}"
2021-03-11 13:08:18,608 ERROR pid=13216 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/ta_ms_loganalytics/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/log_analytics.py", line 96, in collect_events
input_module.collect_events(self, ew)
File "$splunkhome$/etc/apps/TA-ms-loganalytics/bin/input_module_log_analytics.py", line 86, in collect_events
for i in range(len(data["tables"][0]["rows"])):
UnboundLocalError: local variable 'data' referenced before assignment

Am i hitting any limitation ? if so, any way to overcome this ?

any suggestions appreciated... @jkat54

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

sahilyahiya — Tue, 16 Mar 2021 09:01:00 GMT

Even I am facing same issue.

response="{"error":{"message":"Response size too large","code":"ResponseSizeError","correlationId":"XXX","innererror":

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

JimboSlice — Wed, 17 Mar 2021 08:35:52 GMT

Hi I'm getting the exact same issue, it started with one workspace a couple of weeks ago now i have 5 of the 6 of them faulting with the error.

I believe there is a 67 MB limit on the response size.

I've trimmed back the time from 300 seconds down to 120 seconds and still get the same error.

We're using generic "search *" queries to get all the tables,

I'm meeting with devops this afternoon to go over it.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

jaihingorani — Wed, 17 Mar 2021 08:40:49 GMT

exactly... reducing the interval doesn't help, as its some kind of response limit it is hitting @JimboSlice do comment here if you are able to trace it out anything... much appreciated!!

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

JimboSlice — Wed, 17 Mar 2021 09:04:26 GMT

Oh yeah i sure will, i think if devops confirm that nothing extra has been added to the LAWS (tables or bulks in to the tables) then we will open a support ticket with azure.

There doesn't seem to be any buffer settings under the hood and from what ive read elsewhere there is a 67MB limit on the azure side, devops originally told me this a few weeks ago and ive seen this on the web on a powerBI forum, but why this is suddenly happening to us all at the same time arouses suspicion.

Azure logs are the worst logs on the planet, constant issues (also event hubs).

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

JimboSlice — Thu, 18 Mar 2021 08:42:27 GMT

We have sent this on to our azure support team to get to the bottom of it, it appeared yesterday that when running queries in the management console for LAWS, this same error came up and we think something has changed on the azure side.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

jaihingorani — Thu, 18 Mar 2021 11:15:06 GMT

right, noticed the same while querying at LA, will reach out to our counterparts and see, will keep posted.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

raja_mta — Thu, 17 Jun 2021 19:12:07 GMT

I'm also having the same error, did you guys able to fix it? @jkat54 can you please help us here?

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

jkat54 — Fri, 18 Jun 2021 01:03:04 GMT

Hey first off, nice work everyone!

I'd love to help but I don't have a lab to test this in anymore. Certainly does sound something changed on their end, but without a lab, I can't develop a solution.

You can use the contact the developer button on splunkbase to email me and we can discuss your options if you like.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

jaihingorani — Fri, 18 Jun 2021 05:59:18 GMT

Hi @raja_mta , i was able to make it work by increasing the interval for one of the input, as interval is set as 300 seconds for most of my inputs, however since few weeks i am again facing the same issue with another input, but increasing the interval is not helping now, will let you know if i am able to make it work by any other way.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

raja_mta — Mon, 01 Nov 2021 17:40:00 GMT

Hi @jkat54 ,

I sent an email to you on this. can you please respond.

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

raja_mta — Thu, 02 Dec 2021 15:09:15 GMT

Hi @JimboSlice ,

By any chance do you have any update from Azure side?

Re: TA-ms-loganalytics data ingestion issue for few sourcetypes

jaihingorani — Mon, 13 Dec 2021 13:46:20 GMT

here goes another input, but it got stopped with different limit.

"innererror":{"code":"ResponseSizeError","message":"Maximum response size of 100000000 bytes exceeded. Actual response Size is 103052502 bytes."

now the limit says it has exceeded 100MB, can anyone explain ?