Getting Data In

Splunk Universal Forwarder through a proxy to Splunk Cloud?

Engager

I did a search and found some older answers that gave the impression that this wasn't possible, but I thought I would ask to see if anything has changed.

My use case is that we are in our own Amazon VPC and want to forward some logs to our Splunk Cloud instance. However, the machines in the various subnets need to go through a proxy to access anything outside of the VPC.

Is there a setting somewhere that can tell the forwarder to connect to Splunk Cloud through a proxy?

0 Karma

Splunk Employee
Splunk Employee

While forwarder-to-indexer traffic can be wrapped in SSL, it's not technically an HTTP connection, and therefore won't properly traverse a web proxy.

The 2 ways I know how to accomplish this are as follows:

  1. Use an intermediate forwarder (generally within a DMZ). Internal hosts have access to this host, and send their logs to the IMF. That host has outbound access to the Cloud stack.
  2. Use a SOCKS v5 Proxy

If you wish to secure your forwarder-to-indexer traffic behind a proxy, note that as of 6.3, Splunk supports the use of SOCKS v5 proxies for forwarder-to-indexer traffic. Details are available on-line at:

http://docs.splunk.com/Documentation/Splunk/6.6.3/Forwarding/ConfigureaforwardertouseaSOCKSproxy

0 Karma

Ultra Champion

Keep in mind that with option 1, you are creating a single point of failure and are limiting the spray of data from many universal forwarders, down to one intermediate forwarder. The result is that the data is less distributed on the indexes because the single forwarder will auto load balance but in chunks. Always better to have many endpoints sending their respective chunks to indexers thereby producing a more random (less serial) spray of data.

This matters because when you search the data, you want it to load from many indexers in parallel so it'll be fast. If a chunk of the data is all on the same indexer, you are limited in search speed by that indexer's ability to get the data back.

Example: Imagine trying to get a 10GB file from a single host, vs 1GB files from 10 hosts. The bottleneck is reading from the host (not network), and as such, the 1GB from 10 hosts is going to me like 10x faster.

0 Karma

Splunk Employee
Splunk Employee

There is no internal proxy setting for Splunk itself (although ES has a modular input for the Threatlists that allows for a proxy setting.) Instead you should be configuring your proxy at the OS level. Both *nix and Windows have this feature..

Here's One Example : http://www.cyberciti.biz/faq/linux-unix-set-proxy-environment-variable/

0 Karma

Communicator

Has there been any progress on this issue in the recent times. I am trying to do a similar thing and not able to send the data through the proxy.

0 Karma

Contributor

Hi ... sorry not much has changed on this front.

"Is there a setting somewhere that can tell the forwarder to connect to Splunk Cloud through a proxy"? I don't think so /No

As must have been explained in the earlier answers..... typically proxy connections are only for http requests. Your forwarder needs to connect over TCP on specific port to send the data... this may not be http. If the objective is to get the data into splunk cloud ... it will have be be designed and setup in collaboration with the network security and AWS teams. Eg: Setting up some standard servers as intermediate forwarders in your VPC and opening them up at the firewall might help.