Deployment Architecture

UF can't perform a handshake with DS that's behind an Apache reverse proxy

FoolMoon
Engager

In our current setup we have a private network with several hosts that have UFs installed, as well as separate hosts for a search head, indexer and a Splunk Deployment server. Since All servers where Splunk UFs are installed are in the same private network, they have simply been set up to use the private IP of the deployment server in deploymentclient.conf.

We also have a separate host where a reverse proxy has been configured, using Apache. This host is the only server that has a public IP. The reverse proxy is used so that we can access the web UIs of Splunk search head and deployment server.

Until now this setup has been working well, but now I have to add another UF that is outside of the private network. To do this, I have added another configuration file to the reverse proxy that looks like this:

Listen 8089 https
<VirtualHost *:8089>
    ServerName ds.oursplunk.com

    ProxyPass "/"  "https://{ds_private_ip}:8089/"
    ProxyPassReverse "/"  "https://{ds_private_ip}:8089/"

    SSLEngine on
    SSLProxyEngine on
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLCertificateFile "/etc/letsencrypt/live/ds.oursplunk.com/cert.pem"
    SSLCertificateKeyFile "/etc/letsencrypt/live/ds.oursplunk.com/privkey.pem"
    SSLCertificateChainFile "/etc/letsencrypt/live/ds.oursplunk.com/fullchain.pem"
</VirtualHost>

The configuration itself seems to be working fine, as I can successfully connect to https://ds.oursplunk.com:8089/ with a curl command from the server where the new UF is installed.

However after adding ds.oursplunk.com:8089 to deploymentclient.conf of the new UF, it can't perform a handshake. The most relevant part from splunkd.log of the UF seems this:

10-04-2017 05:33:37.379 -0400 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
10-04-2017 05:33:46.755 -0400 INFO  HttpPubSubConnection - SSL connection with id: connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}
10-04-2017 05:33:46.759 -0400 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /services/broker/channel/subscribe/connection_{rev_proxy_private_ip}_8089_{rev_proxy_internal_dsn}_{uf_hostname}_{uf_client_name}/tenantService/handshake/reply/{uf_hostname}/{uf_client_name} was not found on this server.</p>\n</body></html>\n

Where {rev_proxy_private_ip} is the private IP of the reverse proxy server; {rev_proxy_internal_dsn} is the internal DNS of the reverse proxy (since we're hosting everything on AWS, it's the one that looks like ip-XX-XX-XXX-XX.aws-region.compute.internal); {uf_hostname} is the hostname of the server with UF; {uf_client_name} is the client name configured in deploymentclient.conf.

So it seems that when the connection id is created, it's kind of mixing things from the UF server and the rev proxy server.
Since I'm not that experienced with web servers, I haven't been able to solve this.

Has anyone encountered a problem like this? Any suggestions for solving it?

secrecys
Explorer

same issue here

0 Karma

jsrobard
Explorer

I think you might be running into issues trying to forward non-http traffic through an http proxy. Proxies are a bit out of my area so consider this answer a best-effort from a non-expert.

If the UF is using https and operates strictly through the REST API then I would expect it to work with a web proxy. If the UF uses some other application protocol, or there is something weird about how the UF uses https, then an http proxy would balk at the request because it could not parse what the UF is sending. I would expect a well-behaved proxy would return an HTTP error code, probably something in the 400-500 error code range. That might show up in the UF logs.

I do not know about Apache but I believe nginx can act as a TCP proxy. That approach might work, but it's not something I've used or researched in depth. I'd talk to your local proxy administrator for some expert information.

If you have the option, I would suggest skipping the proxy and manage the UF manually.

You'll also need to have a solution to cover log forwarding from UF to your internal Splunk servers (which happens using a proprietary protocol). I would recommend having a conversation with your networking folks about what options you have to pass the traffic Splunk needs to your private network. A NAT would be one option.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...