I have recently been playing around with the REST API application and the streaming twitter feed and have come across an odd issue. After a lot of troubleshooting, it appears that everything works fine, only when you have a search term that brings back a high volume of events continuously. However, if you use a search term that looks for the odd event here or there, it seems there are python errors in splunkd with SSL timeouts etc.
This is my base configuration (minus the oauth stuff):
[rest://Twitter] auth_type = oauth1 endpoint = https://stream.twitter.com/1.1/statuses/filter.json host = TwitterAPI http_method = GET index = main index_error_response_codes = 1 response_type = json sourcetype = tweets streaming_request = 1 url_args = track=cold,splunk^stall_warnings=true delimiter = ^ disabled = 0
So the above configuration works fine as tracking the word 'cold' brings back a pretty hefty number of events. However, if i remove the word cold and just use 'splunk' which has a far lower tweet rate than the word cold, I get java errors in Splunkd after about 30 secs as follows.
All of the below errors are preceded by:
ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\rest_ta\bin\rest.py"": ssl.SSLError: The read operation timed out return self._sslobj.read(len) File "C:\Program Files\Splunk\Python-2.7\Lib\ssl.py", line 162, in read return self.read(buflen) File "C:\Program Files\Splunk\Python-2.7\Lib\ssl.py", line 243, in recv data = self._sock.recv(self._rbufsize) File "C:\Program Files\Splunk\Python-2.7\Lib\socket.py", line 476, in readline line = self.fp.readline(_MAXLINE + 1) File "C:\Program Files\Splunk\Python-2.7\Lib\httplib.py", line 585, in _read_chunked return self._read_chunked(amt) File "C:\Program Files\Splunk\Python-2.7\Lib\httplib.py", line 543, in read File "C:\Program Files\Splunk\etc\apps\rest_ta\bin\requests-2.0.0-py2.7.egg\requests\packages\urllib3\response.py", line 174, in read File "C:\Program Files\Splunk\etc\apps\rest_ta\bin\requests-2.0.0-py2.7.egg\requests\packages\urllib3\response.py", line 225, in stream File "C:\Program Files\Splunk\etc\apps\rest_ta\bin\requests-2.0.0-py2.7.egg\requests\models.py", line 572, in generate File "C:\Program Files\Splunk\etc\apps\rest_ta\bin\requests-2.0.0-py2.7.egg\requests\models.py", line 602, in iter_lines for line in r.iter_lines(): File "C:\Program Files\Splunk\etc\apps\rest_ta\bin\rest.py", line 465, in do_run do_run()
The message at the bottom is where I hit save to update the url_args with just track=splunk. The error messages all appear after about 30 secs of hitting save on the rest inputs. After the SSL error, i think it just bugs out and does nothing going forward.
You need to get access for the access of twitter. Twitter created a curl like tool called twurl
twurl authorize --consumer-key key --consumer-secret secret
After the request with twurl I can use it with Splunk.
There is antimeout setting that is some what hidden in splunkhome/etc/apps/youraddonapp/bin/youraddon_app/cloudconnectlib/core/defaults.py
The default 'timeout' setting there is two minutes. Change it to something longer.
The default setting was causing my add_on to timeout (SSLError: ('The read operation timed out')) when pulling (GET) huge REST API data in to Splunk.
So it seems if the 'Request Timeout' is set greater than the 30 second default and such that it is of a length greater than the time taken for at least one low volume tweet to come in, then the errors do not appear and it doesn't bug out. It looks as though there is a 'Backoff Time' that implies it would retry after an error but this doesn't seem to be the case here. Anyhow - point is that it works if you extend the timeout period.
I know this is an old topic, but I have the exact same issue with the Splunk Add-on for Tenable.
Fetching from Nessus API times out if it takes longer than 30 seconds, with the same message "SSLError: The read operation timed out".
Where exactly do I set the timeout value, to override the default 30sec, for a modular input that uses Python?