Hi ,
I have a Microsoft Log Analytics Add-on on a heavy forwarder with interval as 60 sec and lag time as 15 min.
Everything works fine till I get below errors-
Query:- index=_internal ERROR sourcetype="ta:ms:loganalytics:log"
Output:-
2018-10-10 08:13:27,405 ERROR pid=10992 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\modinput_wrapper\base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\log_analytics.py", line 96, in collect_events
input_module.collect_events(self, ew)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\input_module_log_analytics.py", line 72, in collect_events
response = requests.post(uri,json=search_params,headers=headers)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\sessions.py", line 641, in send
r.content
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 781, in content
self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "F:\Splunk\etc\apps\TA-ms-loganalytics\bin\ta_ms_loganalytics\requests\models.py", line 706, in generate
raise ChunkedEncodingError(e)
ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2018-10-10 08:19:04,789 ERROR pid=7208 tid=MainThread file=base_modinput.py:log_error:307 | OMSInputName="omslog" status="502" step="Post Query" response="<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>
"
Once this error comes, OMS data flow gets stopped until I re-enable input. and when I re-enable input it again starts flowing.
Can any one help me? What will be the issue causing data to stopped and not reconnecting again once issue is resolved?
Hi @ips_mandar, Is your issue resolved yet? I am facing a similar issue with the splunk reporting Add-on for Office365
It appears you have a proxy server or load balancer (nginx) configured for this splunk devices outbound connections and it's causing the issue:
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
This may not directly answer your question, but I have noticed this behavior in the past with another Microsoft Add-On that uses modinputs. It seems that if there is an error in the execution, it seems to be removed from the scheduling, which would seem to be a bug in Splunk itself. My guess is that if the current input is still running, it will skip until the next run and the failure causes Splunk not to register that there was a failure. Next time there is a failure, try going to the modinputs api URL below and see if it still thinks it is running or not:
hi @ips_mandar,
did this help you answer you question? If so, please approve it so other users can learn from it. Thanks for posting!
Hi @mstjohn_splunk ,
My issue is not yet resolved..
Hi @ips_mandar,
Is your issue resolved yet? I am facing a similar issue with the Splunk reporting Add-on for Office 365 and looking for help.
Currently it is running and it shows-
exit status description exited with code 0
time opened 2018-10-16T11:04:20+0200
total bytes 28587367
I will keep watching once data flow gets stopped...
but is there any solution to avoid this problem?
Not sure why the comment isn't showing up, but I saw your reply that the input was now gone. This definitely seems like a bug, either with the modinputs or with the way this app is designed. I would contact support and file a bug report.
actually i checked on my search head regarding input so might be it won't show up then i got to know i need to check on HF but till that time I have re-enabled input ...so I need to check again on HF when data will get stopped.
Now I checked after data get stop but it will not conclude if input is stopped or not because it looks same -
exit status description exited with code 0
time opened 2018-10-16T19:10:05+0200
total bytes 17842283