topic Re: Does this app support fetching data via proxy? in All Apps and Add-ons

Does this app support fetching data via proxy?

rvaglid — Tue, 29 Sep 2020 18:23:14 GMT

I can not find any information on Splunkbase or in the .conf-files regarding support for using proxy for fetching data.

I notice there is a function "remove_http_proxy_env_vars" in the python code, so I assume if the environment variable http_proxy is set, it is specifically removed when fetching data?

def remove_http_proxy_env_vars():
    for k in ("http_proxy", "https_proxy"):
        if k in os.environ:
            del os.environ[k]
        elif k.upper() in os.environ:
            del os.environ[k.upper()]

Is there a way around this? Is support for proxy in the works for the next version? We have strict rules, forcing us to use proxy when connecting to the Internet.

Cheers
Rolf

Re: Does this app support fetching data via proxy?

jconger — Tue, 29 Sep 2020 18:51:22 GMT

Not yet. But, in the meantime, you can edit the input_module_MS_AAD_audit.py and input_module_MS_AAD_signins.py files. These files do the work for the different inputs. The Python requests library is utilized to make calls to the Microsoft APIs, and this library does support proxy severs. Look for the following lines of code in the files:

header = {'Authorization':'Bearer ' + access_token}
r = requests.get(url,headers=header)

Change it to:

header = {'Authorization':'Bearer ' + access_token}
proxies = {
    'http': 'http://proxy_server_address:proxy_server_port',
    'https': 'http://proxy_server_address:proxy_server_port',
}
r = requests.get(url,proxies=proxies,headers=header)

Re: Does this app support fetching data via proxy?

rvaglid — Tue, 29 Sep 2020 18:53:53 GMT

Thanks Jason, seems to work fine. We're running SSL-inspection on the proxies so I received the following, even tho the CA certificates are present in the OS cert store (RedHat 7.4):
2018-04-06 12:20:58,356 ERROR pid=8322 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 68, in collect_events
input_module.collect_events(self, ew)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 75, in collect_events
r = requests.get(url,proxies=proxies,headers=header)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/requests/api.py", line 70, in get
return request('get', url, params=params, **kwargs)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/requests/adapters.py", line 497, in send
raise SSLError(e, request=request)
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:676)

I did a bit of digging and changed the following in ta_ms_aad/modinput_wrapper/base_modinput.py to point to the correct CA store:
return self.rest_helper.send_http_request(url=url, method=method, parameters=parameters, payload=payload,
headers=headers, cookies=cookies, verify='/etc/pki/tls/certs/ca-bundle.crt', cert=cert,
timeout=timeout, proxy_uri=self._get_proxy_uri() if use_proxy else None)

Re: Does this app support fetching data via proxy?

pclemo_bw — Fri, 01 Jun 2018 06:26:09 GMT

Is there an ETA of this proxy functionality being added to this TA?

Re: Does this app support fetching data via proxy?

hatalla — Tue, 29 Sep 2020 22:19:52 GMT

hey guys,

I deployed the TA on our Splunk deployment and it is behind a firewall. So I edited both input_module_MS_AAD_audit.py and input_module_MS_AAD_signins.py as such:

proxies = {
'http': 'http://my-proxy-server:3128',
'https':'http://my-proxy-server:3128',
}
r = requests.get(url,proxies=proxies,headers=header)
### r = requests.get(url, headers=header)

But I am seeing this error:

12-10-2018 16:37:11.737 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" ERRORcannot concatenate 'str' and 'exceptions.KeyError' objects

12-10-2018 16:37:11.714 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" TypeError: cannot concatenate 'str' and 'exceptions.KeyError' objects

12-10-2018 16:37:11.714 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" header = {'Accept':'application/json', 'Authorization':'Bearer ' + access_token}

12-10-2018 16:37:11.714 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" File "/splunk/app/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 90, in get_audit_events

12-10-2018 16:37:11.714 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" audit_events = get_audit_events(helper, access_token, url, max_records)

12-10-2018 16:37:11.714 -0500 ERROR ExecProcessor - message from "python /splunk/app/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" input_module.collect_events(self, ew)

I've plugged the Azure clientID and secretKey in the TA's config screen. What am I missing?

Thanks.