All Apps and Add-ons

How about proxy in Google Spreadsheets?

YotaLab
Engager

We have splunk server and http proxy around. All works fine, but when we try to use Google Spreadsheets Data input, we have:

2018-03-26 20:47:38,686 ERROR A general exception was thrown when executing the import
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/google_drive/bin/google_spreadsheet.py", line 201, in import_file
    google_lookup_sync = GoogleLookupSync(client_email, private_key, logger=self.logger)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 62, in __init__
    self.gspread_client = self.make_client(private_key, client_email)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 87, in make_client
    return gspread.authorize(credentials)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 326, in authorize
    client.login()
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 85, in login
    self.auth.refresh(http)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 633, in refresh
    self._refresh(http.request)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 842, in _refresh
    self._do_refresh_request(http_request)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/oauth2client/client.py", line 874, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1611, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/splunk/lib/python2.7/site-packages/httplib2/__init__.py", line 1278, in _conn_request
    raise ServerNotFoundError("Unable to find the server at %s" % conn.host)
ServerNotFoundError: Unable to find the server at accounts.google.com

I think we should set proxy use, but where? Process of key identification we completed correctly at first atempt, not problem with it. I think we have problem with resolving from python. But from server - not problem:
splunk@splunk:/root$ dig accounts.google.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> accounts.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11063
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 9

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1024
;; QUESTION SECTION:
;accounts.google.com.       IN  A

;; ANSWER SECTION:
accounts.google.com.    7187    IN  A   64.233.162.84

What we should do?

Tags (1)
0 Karma

LukeMurphey
Champion

I'm not sure this is going to work. gspread (the library I am using to access Google's API) reportedly doesn't support proxies (see https://stackoverflow.com/questions/29549701/using-gspread-with-proxy?utm_medium=organic&utm_source=...).

You could try setting up a proxy per the answer here and seeing if it will work: https://answers.splunk.com/answers/59873/how-to-get-splunk-to-work-behind-a-proxy.html

0 Karma

YotaLab
Engager

Thanks for help.

We debugged the script and found that it uses the httplib2 module, which has no parameters for specifying a proxy_info. We have specified the proxy_info parameters here in google_spreadsheet.py and in a similar way for gspread/client.py:

http = httplib2.Http (proxy_info = httplib2.ProxyInfo (httplib2.socks.PROXY_TYPE_HTTP, 'proxy.in.our.net', 3128, proxy_rdns = False), disable_ssl_certificate_validation = True)

This solved our problem, but only after adding proxy_rdns=False parameter.

Now we have other problem:

2018-04-25 13:11:37,358 INFO Previous run was too far in the past (gap=613.1967689990997) and thus some executions of the input have been missed (stanza=google_spreadsheet://marketing)
2018-04-25 13:14:12,364 ERROR A general exception was thrown when executing the import
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/google_drive/bin/google_spreadsheet.py", line 211, in import_file
    last_updated = google_lookup_sync.import_to_lookup_file(lookup_name, None, None, spreadsheet_title, worksheet_name, session_key, create_if_non_existent=False)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 310, in import_to_lookup_file
    return self.import_to_lookup_file_full_path(destination_full_path, namespace, owner, google_spread_sheet_name, worksheet_name, session_key, create_if_non_existent, lookup_name=lookup_name)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 328, in import_to_lookup_file_full_path
    google_spread_sheet = self.open_google_spreadsheet(google_spread_sheet_name)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/__init__.py", line 112, in open_google_spreadsheet
    google_spread_sheet = self.gspread_client.open(title)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 134, in open
    feed = self.get_spreadsheets_feed()
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/client.py", line 220, in get_spreadsheets_feed
    r = self.session.get(url)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/httpsession.py", line 79, in get
    return self.request('GET', url, **kwargs)
  File "/opt/splunk/etc/apps/google_drive/bin/google_drive_app/gspread/httpsession.py", line 71, in request
    self.connections[uri.scheme+uri.netloc].request(method, url, data, headers=request_headers)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/opt/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/opt/splunk/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/opt/splunk/lib/python2.7/httplib.py", line 1255, in connect
    HTTPConnection.connect(self)
  File "/opt/splunk/lib/python2.7/httplib.py", line 821, in connect
    self.timeout, self.source_address)
  File "/opt/splunk/lib/python2.7/socket.py", line 575, in create_connection
    raise err
error: [Errno 99] Cannot assign requested address

Can you help with it? Is this problem a reason of "Previous run was too far in the past"? Thanks.

0 Karma

YotaLab
Engager

Maybe @LukeMurphey can help?

0 Karma

YotaLab
Engager

From splunk python installation i can GET content with httplib2:

splunk@splunk:/opt/splunk/bin$ ./python
Python 2.7.13 (default, Dec  1 2017, 09:46:10)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib2
>>> h = httplib2.Http(".cache")
>>> resp, content = h.request("http://accounts.google.com", "GET")
>>> print content

<!DOCTYPE html>
<html lang="en">
  <head>
  <meta charset="utf-8">
  <meta content="width=300, initial-scale=1" name="viewport">
  <meta name="google-site-verification" content="LrdTUW9psUAMbh4Ia074-BPEVmcpBxF6Gwf0MSgQXZs">
  <title>Sign in - Google Accounts</title>
  <style>
  @font-face {
  font-family: 'Open Sans';
  font-style: normal;
  font-weight: 300;
  src: local('Open Sans Light'), local('OpenSans-Light'), url(//fonts.gstatic.com/s/opensans/v15/mem5YaGs126MiZpBA-UN_r8OUuhs.ttf) format('truetype');
}
.........../
0 Karma

Anam
Community Manager
Community Manager

Hi @YotaLab

My name is Anam, and I am the Community Content Specialist for Splunk Answers.

This question that you posted has an IP address exposing the hostname and the organization.

I highly recommend you delete that question or let me know and I can delete it for you and post a new question masking the IP address. We cannot make edits to the post because they will be visible in the revision history for everyone to view. We’ve had companies request content be taken down because information like host names, IPs, and other sensitive information was shared in our public forum, so we just want to make sure members of the Splunk community cover their bases.

Thanks, and let me know if you have any questions.

0 Karma

YotaLab
Engager

What you mean? There is only accouns.google.com ip address, not mine.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...