All Apps and Add-ons

Why is Microsoft Log Analytics Add-on(Formerly Know as OMS) data getting stopped after 502 Response code?

Builder

Hi ,

I have a Microsoft Log Analytics Add-on on a heavy forwarder with interval as 60 sec and lag time as 15 min.

Everything works fine till I get below errors-
Query:- index=_internal ERROR sourcetype="ta:ms:loganalytics:log"
Output:-

2018-10-10 08:13:27,405 ERROR pid=10992 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
 Traceback (most recent call last):
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events
     self.collect_events(ew)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events
     input_module.collect_events(self, ew)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 72, in collect_events
     response = requests.post(uri,json=search_params,headers=headers)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 110, in post
     return request('post', url, data=data, json=json, **kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 56, in request
     return session.request(method=method, url=url, **kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 488, in request
     resp = self.send(prep, **send_kwargs)
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 641, in send
     r.content
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 781, in content
     self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
   File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 706, in generate
     raise ChunkedEncodingError(e)
 ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
 2018-10-10 08:19:04,789 ERROR pid=7208 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="omslog" status="502" step="Post Query" response="<html>
 <head><title>502 Bad Gateway</title></head>
 <body bgcolor="white">
 <center><h1>502 Bad Gateway</h1></center>
 <hr><center>nginx</center>
 </body>
 </html>
 "

Once this error comes, OMS data flow gets stopped until I re-enable input. and when I re-enable input it again starts flowing.
Can any one help me? What will be the issue causing data to stopped and not reconnecting again once issue is resolved?

New Member

Hi @ips_mandar, Is your issue resolved yet? I am facing a similar issue with the splunk reporting Add-on for Office365

0 Karma

SplunkTrust
SplunkTrust

It appears you have a proxy server or load balancer (nginx) configured for this splunk devices outbound connections and it's causing the issue:

 <head><title>502 Bad Gateway</title></head>
 <body bgcolor="white">
 <center><h1>502 Bad Gateway</h1></center>
 <hr><center>nginx</center>
 </body>
0 Karma

Explorer

This may not directly answer your question, but I have noticed this behavior in the past with another Microsoft Add-On that uses modinputs. It seems that if there is an error in the execution, it seems to be removed from the scheduling, which would seem to be a bug in Splunk itself. My guess is that if the current input is still running, it will skip until the next run and the failure causes Splunk not to register that there was a failure. Next time there is a failure, try going to the modinputs api URL below and see if it still thinks it is running or not:

https://localhost:8089/services/admin/inputstatus

0 Karma

Splunk Employee
Splunk Employee

hi @ips_mandar,

did this help you answer you question? If so, please approve it so other users can learn from it. Thanks for posting!

0 Karma

Builder

Hi @mstjohn_splunk ,
My issue is not yet resolved..

0 Karma

New Member

Hi @ips_mandar,

Is your issue resolved yet? I am facing a similar issue with the Splunk reporting Add-on for Office 365 and looking for help.

0 Karma

Builder

Currently it is running and it shows-
exit status description exited with code 0
time opened 2018-10-16T11:04:20+0200
total bytes 28587367
I will keep watching once data flow gets stopped...
but is there any solution to avoid this problem?

0 Karma

Explorer

Not sure why the comment isn't showing up, but I saw your reply that the input was now gone. This definitely seems like a bug, either with the modinputs or with the way this app is designed. I would contact support and file a bug report.

0 Karma

Builder

actually i checked on my search head regarding input so might be it won't show up then i got to know i need to check on HF but till that time I have re-enabled input ...so I need to check again on HF when data will get stopped.

0 Karma

Builder

Now I checked after data get stop but it will not conclude if input is stopped or not because it looks same -
exit status description exited with code 0
time opened 2018-10-16T19:10:05+0200
total bytes 17842283

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!