Splunk 6.5 is ignoring https_proxy in splunk-launch.conf

[awmorris]

So Splunk is ignoring the https_proxy settings in my splunk-launch.conf file. The details of how i am positive are shown below:

=========================
== Sanitized Splunk Server Address : 10.1.2.3
== Sanitized DNS Server : 10.4.5.6

== Sanitized Proxy Server : 10.2.3.4

CONTENTS OF SPLUNK.CONF

#   Version 6.4.0  [NOTE: We are currently running 6.5.0 - upgraded from 6.4]

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/home/build/build-home/galaxy

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
SPLUNK_DB=/event_data/var/lib/splunk
#SPLUNK_DB=/home/build/build-home/galaxy/var/lib/splunk

# Splunkd daemon name
SPLUNK_SERVER_NAME=Splunkd

# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER

#Configure proxy servers
https_proxy=https://10.2.3.4:8123

RESULTS OF SPLUNK ENVVARS

PATH=/opt/splunk/bin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:. ; export PATH ; https_proxy=https://10.2.3.4:8123 ; export https_proxy ; SPLUNK_HOME=/opt/splunk ; export SPLUNK_HOME ; SPLUNK_DB=/event_data/var/lib/splunk ; export SPLUNK_DB ; SPLUNK_ETC=/opt/splunk/etc ; export SPLUNK_ETC ; NO_PROXY=127.0.0.1,localhost,\[::1\] ; export NO_PROXY ; SPLUNK_SERVER_NAME=Splunkd ; export SPLUNK_SERVER_NAME ; SPLUNK_WEB_NAME=splunkweb ; export SPLUNK_WEB_NAME ; PYTHONPATH=/opt/splunk/lib/python2.7/site-packages ; export PYTHONPATH ; PYTHONHASHSEED=random ; export PYTHONHASHSEED ; NODE_PATH=/opt/splunk/lib/node_modules ; export NODE_PATH ; LD_LIBRARY_PATH=/opt/splunk/lib ; export LD_LIBRARY_PATH ; OPENSSL_CONF=/opt/splunk/openssl/openssl.cnf ; export OPENSSL_CONF ; LDAPCONF=/opt/splunk/etc/openldap/ldap.conf ; export LDAPCONF

[Started Splunk and navigated to the Apps page... See the TCP dump headers that follow. SPECIFICALLY, notice how it ignores the proxy and goes direct]

FINALLY, THE RESULTS OF TCPDUMP SHOWING THAT IT IGNORES THE HTTPS PROXY AND ATTEMPTS TO TALK DIRECT:

splunk:~ # tcpdump -ni eth0 port 80 or port 443 or port 9090 or port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

22:48:04.377467 IP 10.1.2.3.55305 > 10.4.5.6.53: 65286+ A? apps.splunk.com. (33)
22:48:04.466342 IP 10.4.5.6.53 > 10.1.2.3.55305: 65286 3/4/4 A 52.11.10.173, A[|domain]
22:48:04.466746 IP 10.1.2.3.34473 > 35.160.59.174.443: S 3461648818:3461648818(0) win 14600 <mss 1460,sackOK,timestamp 5600708 0,nop,wscale 7>
22:48:07.473222 IP 10.1.2.3.34473 > 35.160.59.174.443: S 3461648818:3461648818(0) win 14600 <mss 1460,sackOK,timestamp 5601460 0,nop,wscale 7>
22:48:13.489228 IP 10.1.2.3.34473 > 35.160.59.174.443: S 3461648818:3461648818(0) win 14600 <mss 1460,sackOK,timestamp 5602964 0,nop,wscale 7>
22:48:14.478032 IP 10.1.2.3.45932 > 52.32.239.55.443: S 2434318423:2434318423(0) win 14600 <mss 1460,sackOK,timestamp 5603211 0,nop,wscale 7>
22:48:17.481148 IP 10.1.2.3.45932 > 52.32.239.55.443: S 2434318423:2434318423(0) win 14600 <mss 1460,sackOK,timestamp 5603962 0,nop,wscale 7>
22:48:23.489231 IP 10.1.2.3.45932 > 52.32.239.55.443: S 2434318423:2434318423(0) win 14600 <mss 1460,sackOK,timestamp 5605464 0,nop,wscale 7>
22:48:24.489314 IP 10.1.2.3.39608 > 52.11.10.173.443: S 1004902738:1004902738(0) win 14600 <mss 1460,sackOK,timestamp 5605714 0,nop,wscale 7>
22:48:27.497139 IP 10.1.2.3.39608 > 52.11.10.173.443: S 1004902738:1004902738(0) win 14600 <mss 1460,sackOK,timestamp 5606466 0,nop,wscale 7>
22:48:33.505232 IP 10.1.2.3.39608 > 52.11.10.173.443: S 1004902738:1004902738(0) win 14600 <mss 1460,sackOK,timestamp 5607968 0,nop,wscale 7>

LAGNIAPPE INFORMATION, if I go to the command prompt and type "curl https://apps.splunk.com" I get a response (albeit the page moved html, but point is, it works through the proxy).

So what gives? I've tried setting http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY. I've tried with and without the "http://" and "https://" in front of the proxy and I still have no luck.

Any ideas?

[awmorris · Accepted solution]

So, support answered my question and i did a subsequent face palm. I followed the documentation located here: http://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/Specifyaproxyserver

only to learn that i need to actually READ the entirety of the documentation.... specifically the part that says:

"Note: The App Manager is not supported for use with a proxy server, if you use a proxy server with Splunk Web, you must download and update apps manually."

Fault is mine for not reading the documentation ENTIRELY.

[rsennett_splunk]

I can't explain why this is missing from the splunk-launch.conf (although I will attend to that next)

but what you want is here:
https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Specifyaproxyserver

The part you missed was:
**

It's an environmental Variable as are
the rest of the settings... and those
are UPPER CASE.

**

doc linked above reproduced below with emphasis on what's been missed:

Use Splunk Web with a proxy server
When Splunk Web is located behind a proxy server, users may have trouble with Splunk Web links that access the Splunk website. For example, some Splunk Web pages link directly to the download site for Splunk apps and many "learn more" links will take you to the online documentation.

To resolve this, simply set the HTTP_PROXY environment variable. For permanent results, you can specify the setting in the splunk-launch.conf configuration file, located in $SPLUNK_HOME/etc/ on *nix systems and %SPLUNK_HOME%\etc\ on Windows.

Note: The App Manager is not supported for use with a proxy server, if you use a proxy server with Splunk Web, you must download and update apps manually.

In splunk-launch.conf, add this attribute/value pair:

HTTP_PROXY = <IP address or host name>:<port number>

HTTP_PROXY = 10.1.8.11:8787 Important: If your proxy server only handles HTTPS requests, you must use the following attribute/value pair:

HTTPS_PROXY = <IP address or host name>:<port number>

HTTPS_PROXY = 10.1.8.11:8888

---

Try that...

[awmorris]

I didn't miss it. In my original post, i ended with "I've tried setting http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY. I've tried with and without the "http://" and "https://" in front of the proxy and I still have no luck"

However, I reverted back to the format that matches EXACTLY the documentation. The last 3 lines of my splunk-launch.conf now look like:

#Configure proxy servers
HTTP_PROXY=10.2.3.4:9090
HTTPS_PROXY=10.2.3.4:9090

The results of TCPDUMP after restarting Splunk and navigating to "Browse more apps" are:

splunk:~ # tcpdump -ni eth0 port 80 or port 443 or port 9090 or port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
13:33:10.876537 IP 10.1.2.3.47116 > 10.4.5.6.53: 2173+ A? apps.splunk.com. (33)
13:33:10.979854 IP 10.4.5.6.53 > 10.1.2.3.47116: 2173 3/4/4 A 35.160.59.174, A[|domain]
13:33:10.980551 IP 10.1.2.3.53083 > 35.160.59.174.443: S 898627335:898627335(0) win 14600 <mss 1460,sackOK,timestamp 18860940 0,nop,wscale 7>
13:33:13.988440 IP 10.1.2.3.53083 > 35.160.59.174.443: S 898627335:898627335(0) win 14600 <mss 1460,sackOK,timestamp 18861692 0,nop,wscale 7>
13:33:20.004435 IP 10.1.2.3.53083 > 35.160.59.174.443: S 898627335:898627335(0) win 14600 <mss 1460,sackOK,timestamp 18863196 0,nop,wscale 7>
13:33:20.983695 IP 10.1.2.3.36356 > 52.32.239.55.443: S 1462765147:1462765147(0) win 14600 <mss 1460,sackOK,timestamp 18863440 0,nop,wscale 7>
13:33:23.988678 IP 10.1.2.3.36356 > 52.32.239.55.443: S 1462765147:1462765147(0) win 14600 <mss 1460,sackOK,timestamp 18864192 0,nop,wscale 7>
13:33:30.004435 IP 10.1.2.3.36356 > 52.32.239.55.443: S 1462765147:1462765147(0) win 14600 <mss 1460,sackOK,timestamp 18865696 0,nop,wscale 7>
13:33:30.990566 IP 10.1.2.3.58230 > 52.11.10.173.443: S 4256274921:4256274921(0) win 14600 <mss 1460,sackOK,timestamp 18865942 0,nop,wscale 7>
13:33:33.024356 IP 10.1.2.3.59211 > 10.4.5.6.53: 39125+ A? splunkbase.splunk.com. (39)
13:33:33.025707 IP 10.1.2.3.52372 > 10.4.5.6.53: 62603+ A? splunkbase.splunk.com. (39)
13:33:33.391981 IP 10.4.5.6.53 > 10.1.2.3.59211: 39125 3/4/4 A 52.32.239.55,[|domain]
13:33:33.392004 IP 10.4.5.6.53 > 10.1.2.3.52372: 62603 3/4/4 A 52.32.239.55,[|domain]
13:33:33.392239 IP 10.1.2.3.46665 > 10.4.5.6.53: 7514+ A? splunkbase.splunk.com. (39)
13:33:33.392386 IP 10.1.2.3.42395 > 10.4.5.6.53: 64154+ A? splunkbase.splunk.com. (39)
13:33:33.394517 IP 10.4.5.6.53 > 10.1.2.3.42395: 64154 3/4/4 A 35.160.59.174,[|domain]
13:33:33.394742 IP 10.1.2.3.58232 > 52.11.10.173.443: S 83125883:83125883(0) win 14600 <mss 1460,sackOK,timestamp 18866543 0,nop,wscale 7>
13:33:33.480372 IP 10.4.5.6.53 > 10.1.2.3.46665: 7514 3/4/4 A 35.160.59.174,[|domain]
13:33:33.480733 IP 10.1.2.3.36378 > 52.32.239.55.443: S 737828933:737828933(0) win 14600 <mss 1460,sackOK,timestamp 18866565 0,nop,wscale 7>
13:33:33.996433 IP 10.1.2.3.58230 > 52.11.10.173.443: S 4256274921:4256274921(0) win 14600 <mss 1460,sackOK,timestamp 18866694 0,nop,wscale 7>
13:33:36.396424 IP 10.1.2.3.58232 > 52.11.10.173.443: S 83125883:83125883(0) win 14600 <mss 1460,sackOK,timestamp 18867294 0,nop,wscale 7>
13:33:36.484426 IP 10.1.2.3.36378 > 52.32.239.55.443: S 737828933:737828933(0) win 14600 <mss 1460,sackOK,timestamp 18867316 0,nop,wscale 7>
13:33:40.004431 IP 10.1.2.3.58230 > 52.11.10.173.443: S 4256274921:4256274921(0) win 14600 <mss 1460,sackOK,timestamp 18868196 0,nop,wscale 7>
13:33:42.404437 IP 10.1.2.3.58232 > 52.11.10.173.443: S 83125883:83125883(0) win 14600 <mss 1460,sackOK,timestamp 18868796 0,nop,wscale 7>
13:33:42.500429 IP 10.1.2.3.36378 > 52.32.239.55.443: S 737828933:737828933(0) win 14600 <mss 1460,sackOK,timestamp 18868820 0,nop,wscale 7>
13:33:45.405709 IP 10.1.2.3.36433 > 52.32.239.55.443: S 2259234196:2259234196(0) win 14600 <mss 1460,sackOK,timestamp 18869546 0,nop,wscale 7>
13:33:45.493719 IP 10.1.2.3.53220 > 35.160.59.174.443: S 571947275:571947275(0) win 14600 <mss 1460,sackOK,timestamp 18869568 0,nop,wscale 7>
13:33:48.412426 IP 10.1.2.3.36433 > 52.32.239.55.443: S 2259234196:2259234196(0) win 14600 <mss 1460,sackOK,timestamp 18870298 0,nop,wscale 7>
13:33:48.500518 IP 10.1.2.3.53220 > 35.160.59.174.443: S 571947275:571947275(0) win 14600 <mss 1460,sackOK,timestamp 18870320 0,nop,wscale 7>

[awmorris]

UPDATE:

I was able to resolve PART of my issue and allow the Splunk App for AWS to reach AWS using the instructions found at http://docs.splunk.com/Documentation/AddOns/released/AWS/Setuptheadd-on

 CONFIGURE A PROXY CONNECTION:
1) Click Splunk Add-on for AWS in your left navigation bar on Splunk Web's home page.
2) Click Configuration in the app navigation bar.
3) Click the Proxy tab.
4) Check the box to enable the proxy connection and fill in the fields required for your proxy.
5) Click Save.

HOWEVER, I still have the issue that I can't browse for more apps from within the Splunk instance because it is still trying to go direct and not honoring the HTTPS_PROXY setting.

[skalliger]

Which linux distribution are you using? Does your proxy require authentication? Since you're talking about enterprise proxy, I guess so. When I set up https_proxy environment variables on diffent operating systems, I often had problems with Ubuntu.

Did you try a syntax like:
http_proxy=http://user:password@proxy:port
and
https_proxy=https://user:password@proxy:port
?

On some distributions (I think on CentOS) you need to write it like ...//user:'password'@... - note the two single ' '.

Skalli

[awmorris]

Good question re: version.
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4

As for authentication, I've worked with the proxy admins to allow traffic via source address. I am confident it works because I can set my environment values and use CURL from the CLI.

I also fall back on my argument that the TCPDUMP shows it never even TRIES to contact the proxy.

[SierraX]

There is no proxy key-value pair in splunk-launch.conf

And there are a few proxy key-value pairs in server.conf

[SierraX]

http://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Splunk-launchconf

Thought that link are also parsed automatically ...

Anyway... The 'Configuration file reference' in Admin section of docs is the first place to search for "what key-value pairs are available in this .conf file"

[awmorris]

Thanks for the answer, but I think the situation is different. I am trying to configure my Splunk instance to use the enterprise proxy to go to the internet. I was following the instructions on this page: https://docs.splunk.com/Documentation/Splunk/6.5.0/Admin/Specifyaproxyserver

All of the proxy settings in the server.conf file are related to clustering.

[SierraX]

Thanks for the link... didn't know that before
forwarded to splunk to update the documentation.