How to make the code work in an Adaptive Respone via the Splunk Add on Builder in dev environment?

[ericl42]

I created an adaptive response via the Splunk Add-on Builder in my dev environment and everything is working fine. When I export the TA to production, the adaptive response shows up correctly,` and all of my predefined items are in there, but the code is not working.

Below is a simplified version of the Python code.

import sys
import json
import requests
import getpass
import requests.packages.urllib3
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

def process_event(helper, *args, **kwargs):

    helper.set_log_level(helper.log_level)

    auth_url = "https://ticketing_system.domain.com"
    login_data = { 'username': 'YYYYY', 'password': 'XXXXXXXX' }

    session = requests.session()

   try:
        r = session.post(auth_url, json=login_data, verify=False)
        web_token = r.text
        r.raise_for_status()
    except Exception as e:
        print(e)
        print(r.text)
        print(r.status_code)
        sys.exit()

    parsed_token = web_token.split('"')[3]

    headers = {'Authorization': 'Bearer ' + parsed_token, 'Content-Type': 'application/json'}

    helper.log_info("Alert action ticketing started.")

    resp = requests.post("https://ticketing_system.domain.com", headers=headers, data=json.dumps(sn_input), verify=False)

    if resp.text:
        print(resp.text)

    helper.addevent(resp.text, sourcetype="ticketing")
    helper.writeevents(index="_internal", host="localhost", source="ticketing")

    return 0

When I run this in prod, I get the following error.

signature="Unexpected error: local variable 'r' referenced before assignment."

So, for some reason, it's not respecting my try statement and the sessions.post. Therefore I simplified that even more and did this.

r = session.post(auth_url, json=login_data, verify=False)
web_token = r.text
r.raise_for_status()

This then gave me a different error.

signature="Unexpected error: HTTPSConnectionPool(host='ticketing_system.domain.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 503 Service Unavailable',)))."

I have confirmed that the Python code outside of Splunk works fine. Firewall rules are opened for direct connections to the ticketing.domain.com site. I thought it may have been picking up on the proxy anyway but when I perform a tcpdump on the server running the adaptive response, I see no outbound connections even occurring.

I also can't seem to get the logging level set to debug that will provide me any additional information on what is truly going on.

Any help would be greatly appreciated as I have no idea why it works in dev but not prod. Note that on dev, I'm using the same box that the add-on builder is running on. I have thought about installing the add-on builder directly to our prod Enterprise Security box but I know that is frowned upon.

[ericl42 · Accepted solution]

What ended up being the issue was that even though I told the add-on builder to not use a proxy, it was still using one from Splunks server.conf and splunk-launch.conf. I added the IPs/hostnames to the no proxy rule thinking that would work, but unfortunately it still didn't.

The final solution to my problem was adding the following lines in my Python code to 100% tell it not to use any proxy.

import os
os.environ['no_proxy'] = '*' 

[lakshman239]

On your add-on builder, do you have setup page with proxy options, if not, can you pls add -that, as that would help to isolate any connection issues between dev and ticketing system and prod and ticketing systems, as that will force the splunk libs to use/not-use proxy libs.

Also I had noticed in some prod servers, additional use of https_proxy and no_proxy settings in the env settings [check by $env | grep proxy]

This could also prevent external connection and you could get time out with Max retries. So, adjust those settings for your ticketing systems and it will work. (you could also try this before adding proxy setup in your add-on if thats quicker]

[ericl42]

Under "Add-on Setup Parameters" I have the "proxy settings" section checked but I don't have anything configured for it (i.e. no text boxes which the information in there).

When I go to the Python code, I don't see any proxy references other than this, which is obviously commented out.

# response is a response object in python requests library
#response = helper.send_http_request("http://www.splunk.com", "GET", parameters=None,
#                                        payload=None, headers=None, cookies=None, verify=True, cert=None, timeout=None, use_proxy=True)

When I log in as the splunk use, I don't see any proxy env variable set.

[lakshman239]

When you build the add-on, in the 'Add-on Setup parameters' if you check 'proxy settings' and then continue to develop your python code and save it. When you restart the instance and go to the 'Add-on setup/config' page, you can see 'enable' check box, proxy type, host, port etc.. [ if you need to use proxy - check the box and enter your proxy hostname and port]

in the process_event code, you can have something like

def check_proxy(helper):
    isProxyEnabled = True
    myproxy_details  = helper.get_proxy()
    if not myproxy_details.get('proxy_url'):
         isProxyEnabled = False
   return isProxyEnabled

and in your send_http.request (....... use_proxy=isProxyEnabled) so, it will use your config based on your env.