We have splunk server and http proxy around. All works fine, but when we try to use Google Spreadsheets Data input, we have:
2018-03-26 20:47:38,686 ERROR A general exception was thrown when executing the import
Traceback (most recent call last):
File "/opt/splunk/etc/apps/google_drive/bin/google_spreadsheet.py", line 201, in import_file
google_lookup_sync = GoogleLookupSync(client_email, private_key, logger=self.logger)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 62, in __init__
self.gspread_client = self.make_client(private_key, client_email)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 87, in make_client
return gspread.authorize(credentials)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 326, in authorize
client.login()
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 85, in login
self.auth.refresh(http)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 633, in refresh
self._refresh(http.request)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 842, in _refresh
self._do_refresh_request(http_request)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 874, in _do_refresh_request
self.token_uri, method='POST', body=body, headers=headers)
File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1611, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1278, in _conn_request
raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
ServerNotFoundError: Unable to find the server at accounts.google.com
I think we should set proxy use, but where? Process of key identification we completed correctly at first atempt, not problem with it. I think we have problem with resolving from python. But from server - not problem:
splunk@splunk:/root$ dig accounts.google.com
; <<>> DiG 9.10.3-P4-Ubuntu <<>> accounts.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11063
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 9
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;accounts.google.com. IN A
;; ANSWER SECTION:
accounts.google.com. 7187 IN A 64.233.162.84
What we should do?
I'm not sure this is going to work. gspread (the library I am using to access Google's API) reportedly doesn't support proxies (see https://stackoverflow.com/questions/29549701/using-gspread-with-proxy?utm_medium=organic&utm_source=...).
You could try setting up a proxy per the answer here and seeing if it will work: https://answers.splunk.com/answers/59873/how-to-get-splunk-to-work-behind-a-proxy.html
Thanks for help.
We debugged the script and found that it uses the httplib2 module, which has no parameters for specifying a proxy_info. We have specified the proxy_info parameters here in google_spreadsheet.py and in a similar way for gspread/client.py:
http = httplib2.Http (proxy_info = httplib2.ProxyInfo (httplib2.socks.PROXY_TYPE_HTTP, 'proxy.in.our.net', 3128, proxy_rdns = False), disable_ssl_certificate_validation = True)
This solved our problem, but only after adding proxy_rdns=False parameter.
Now we have other problem:
2018-04-25 13:11:37,358 INFO Previous run was too far in the past (gap=613.1967689990997) and thus some executions of the input have been missed (stanza=google_spreadsheet://marketing)
2018-04-25 13:14:12,364 ERROR A general exception was thrown when executing the import
Traceback (most recent call last):
File "/opt/splunk/etc/apps/google_drive/bin/google_spreadsheet.py", line 211, in import_file
last_updated = google_lookup_sync.import_to_lookup_file(lookup_name, None, None, spreadsheet_title, worksheet_name, session_key, create_if_non_existent=False)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 310, in import_to_lookup_file
return self.import_to_lookup_file_full_path(destination_full_path, namespace, owner, google_spread_sheet_name, worksheet_name, session_key, create_if_non_existent, lookup_name=lookup_name)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 328, in import_to_lookup_file_full_path
google_spread_sheet = self.open_google_spreadsheet(google_spread_sheet_name)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 112, in open_google_spreadsheet
google_spread_sheet = self.gspread_client.open(title)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 134, in open
feed = self.get_spreadsheets_feed()
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 220, in get_spreadsheets_feed
r = self.session.get(url)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/httpsession.py", line 79, in get
return self.request('GET', url, **kwargs)
File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/httpsession.py", line 71, in request
self.connections[uri.scheme+uri.netloc].request(method, url, data, headers=request_headers)
File "/opt/splunk/lib/python2.7/httplib.py", line 1042, in request
self._send_request(method, url, body, headers)
File "/opt/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
self.endheaders(body)
File "/opt/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
self._send_output(message_body)
File "/opt/splunk/lib/python2.7/httplib.py", line 882, in _send_output
self.send(msg)
File "/opt/splunk/lib/python2.7/httplib.py", line 844, in send
self.connect()
File "/opt/splunk/lib/python2.7/httplib.py", line 1255, in connect
HTTPConnection.connect(self)
File "/opt/splunk/lib/python2.7/httplib.py", line 821, in connect
self.timeout, self.source_address)
File "/opt/splunk/lib/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 99] Cannot assign requested address
Can you help with it? Is this problem a reason of "Previous run was too far in the past"? Thanks.
Maybe @LukeMurphey can help?
From splunk python installation i can GET content with httplib2:
splunk@splunk:/opt/splunk/bin$ ./python
Python 2.7.13 (default, Dec 1 2017, 09:46:10)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> h = httplib2.Http(".cache")
>>> resp, content = h.request("http://accounts.google.com", "GET")
>>> print content
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta content="width=300, initial-scale=1" name="viewport">
<meta name="google-site-verification" content="LrdTUW9psUAMbh4Ia074-BPEVmcpBxF6Gwf0MSgQXZs">
<title>Sign in - Google Accounts</title>
<style>
@font-face {
font-family: 'Open Sans';
font-style: normal;
font-weight: 300;
src: local('Open Sans Light'), local('OpenSans-Light'), url(//fonts.gstatic.com/s/opensans/v15/mem5YaGs126MiZpBA-UN_r8OUuhs.ttf) format('truetype');
}
.........../
Hi @YotaLab
My name is Anam, and I am the Community Content Specialist for Splunk Answers.
This question that you posted has an IP address exposing the hostname and the organization.
I highly recommend you delete that question or let me know and I can delete it for you and post a new question masking the IP address. We cannot make edits to the post because they will be visible in the revision history for everyone to view. We’ve had companies request content be taken down because information like host names, IPs, and other sensitive information was shared in our public forum, so we just want to make sure members of the Splunk community cover their bases.
Thanks, and let me know if you have any questions.
What you mean? There is only accouns.google.com ip address, not mine.