Splunk Python SDK - Causing HTTP 503 (HTTP Too Many Threads) and Socket Errno=110

chrismmckenna — Tue, 29 Sep 2020 16:04:13 GMT

Suggestions for improvement to the Python SDK script implementation are being requested. Would modifying the EXEC_MODE or OUTPUT_MODE to another value help?

I'm am using a Python SDK (splunk-sdk-python-1.6.2) script in the examples directory (search.py) on a heavy forwarder to collect search results from a Splunk Enterprise server, writing them to file, monitoring the file and forwarding to Splunk Cloud.

I've wrapped the search.py script it in a BASH shell script and it is somewhat successfully executing from the splunk user crontab every minute. Initially, it appears data is collected and everything is working fine. However, after a few minutes, I start to receive HTTP Error 503 (too many HTTP threads) and start to get socket timeout errors (errno 110).

Eventually, the host's memory utilization is so high that it is no longer reachable and needs to be rebooted. I can see there a variety of processes spawned, like: kthreadd, ksoftirqd/0, kworker/0:0H and the like.

I know the one minute, repeated execution is a lot and am working with the requestors to change that requirement. In addition, I have asked them to consider forwarding the data directly to Splunk Cloud. In the meantime, I am trying to get a stable implementation working.

The BASH wrapper:

-#Modify this file if you need to change PYTHONPATH, host, port, username or password

SCRIPT_HOME=/opt/splunk/etc/apps/gcs-shippingapi/bin
source $SCRIPT_HOME/gcs-shippingapi-hostcred.cfg

-#The time string can be either a UTC time (with fractional seconds), a relative time specifier (to now) or a formatted time string.

EARLIEST='-2m@m'
LATEST='-1m@m'

-#Execution mode valid values: (blocking | oneshot | normal); default=normal
-#Refer to the following for more information: http://dev.splunk.com/view/python-sdk/SP-CAAAEE5
EXEC_MODE='oneshot'

-#Output mode valid values: (atom | csv | json | json_cols | json_rows | raw | xml); default=xml
OUTPUT_MODE='raw'

SEARCH='search Org=pitneybowes AND Env=prod AND EndpointName= AND responseStatus='

/opt/splunk/bin/python $SCRIPT_HOME/search.py "$SEARCH" --host=$SPLUNK_HOST --port=$PORT --username=$SPLUNK_USERNAME --password=$SPLUNK_PASSWORD --output_mode=$
OUTPUT_MODE --earliest_time=$EARLIEST --latest_time=$LATEST

Cron Error Message #1:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 115, in main(sys.argv[1:]) File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 72, in main service = client.connect(**kwargs_splunk) File "/opt/splunk-sdk-python-1.6.2/splunklib/client.py", line 321, in connect s.login() File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 857, in login cookie="1") # In Splunk 6.2+, passing "cookie=1" will return the "set-cookie" header File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1201, in post return self.request(url, message) File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1221, in request raise HTTPError(response) splunklib.binding.HTTPError: HTTP 503 Too many HTTP threads (628) already running, try again later --

Too many HTTP threads (628) already running, try again later
The server can not presently handle the given request.

Cron Error Message #2:

Traceback (most recent call last):
  File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 115, in 
    main(sys.argv[1:])
  File "/opt/splunk/etc/apps/gcs-shippingapi/bin/search.py", line 72, in main
    service = client.connect(**kwargs_splunk)
  File "/opt/splunk-sdk-python-1.6.2/splunklib/client.py", line 321, in connect
    s.login()
  File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 857, in login
    cookie="1") # In Splunk 6.2+, passing "cookie=1" will return the "set-cookie" header
  File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1201, in post
    return self.request(url, message)
  File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1218, in request
    response = self.handler(url, message, **kwargs)
  File "/opt/splunk-sdk-python-1.6.2/splunklib/binding.py", line 1357, in request
    connection.request(method, path, body, head)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/opt/splunk/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/opt/splunk/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/opt/splunk/lib/python2.7/httplib.py", line 844, in send
      self.connect()
  File "/opt/splunk/lib/python2.7/httplib.py", line 1255, in connect
    HTTPConnection.connect(self)
  File "/opt/splunk/lib/python2.7/httplib.py", line 821, in connect
    self.timeout, self.source_address)
  File "/opt/splunk/lib/python2.7/socket.py", line 575, in create_connection
    raise err
socket.error: [Errno 110] Connection timed out

Re: Splunk Python SDK - Causing HTTP 503 (HTTP Too Many Threads) and Socket Errno=110

sloshburch — Fri, 06 Oct 2017 22:09:12 GMT

"heavy forwarder to collect search results from a Splunk Enterprise server, writing them to file, monitoring the file and forwarding to Splunk Cloud."

This seems like a complete overengineering. We might be able to solve the same thing with native features in Splunk rather than creating new script and the resulting complexities and points of failures.

Let's make sure we have the problem correct: are you trying to collect data from an On Prem splunk and selectively forward to Splunk Cloud?

Are you searching for Org=pitneybowes AND Env=prod AND EndpointName= AND responseStatus= over a minute ago?
The search itself is interesting to me for the following reasons:

Is Org a field? With the leading capitalized O
Is this data always from a particular index?
Are these fields part of the log payload or have you added fields at search time?
Why the AND in between given any collection of items in a search are implicitly AND?
Are you looking for events where the specific text EndpointName= AND responseStatus= are shown? I ask because you may have preferred the search to be responseStatus=* EndpointName=* which means only events where there is a value for those fields.

Anyway, the way I would consider setting this up (in order of least complexity) is to:

Have the forwarders send data to both on prem AND cloud https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Configureforwarderswithoutputs.confd#Data_cloning
Have the indexers on prem index AND forward https://docs.splunk.com/Documentation/SplunkCloud/latest/Forwarding/Configureforwarderswithoutputs.confd#Global_stanza
Have the Heavy Forwarder run the search as a Summary Indexing activity but set up to just forward that index to Cloud https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Usesummaryindexing

Let me know if there's features of the SDK you think are needed and why. I'll use that information to see if there's still out-of-the-box solutions for solving your problem rather than creating a new script and all.

topic Re: Splunk Python SDK - Causing HTTP 503 (HTTP Too Many Threads) and Socket Errno=110 in Splunk Dev

Splunk Python SDK - Causing HTTP 503 (HTTP Too Many Threads) and Socket Errno=110

Re: Splunk Python SDK - Causing HTTP 503 (HTTP Too Many Threads) and Socket Errno=110