Getting Data In

Splunk Universal Forwarder through a proxy to Splunk Cloud?

dustinconrad
Engager

I did a search and found some older answers that gave the impression that this wasn't possible, but I thought I would ask to see if anything has changed.

My use case is that we are in our own Amazon VPC and want to forward some logs to our Splunk Cloud instance. However, the machines in the various subnets need to go through a proxy to access anything outside of the VPC.

Is there a setting somewhere that can tell the forwarder to connect to Splunk Cloud through a proxy?

0 Karma

mnatkin_splunk
Splunk Employee
Splunk Employee

While forwarder-to-indexer traffic can be wrapped in SSL, it's not technically an HTTP connection, and therefore won't properly traverse a web proxy.

The 2 ways I know how to accomplish this are as follows:

  1. Use an intermediate forwarder (generally within a DMZ). Internal hosts have access to this host, and send their logs to the IMF. That host has outbound access to the Cloud stack.
  2. Use a SOCKS v5 Proxy

If you wish to secure your forwarder-to-indexer traffic behind a proxy, note that as of 6.3, Splunk supports the use of SOCKS v5 proxies for forwarder-to-indexer traffic. Details are available on-line at:

http://docs.splunk.com/Documentation/Splunk/6.6.3/Forwarding/ConfigureaforwardertouseaSOCKSproxy

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Keep in mind that with option 1, you are creating a single point of failure and are limiting the spray of data from many universal forwarders, down to one intermediate forwarder. The result is that the data is less distributed on the indexes because the single forwarder will auto load balance but in chunks. Always better to have many endpoints sending their respective chunks to indexers thereby producing a more random (less serial) spray of data.

This matters because when you search the data, you want it to load from many indexers in parallel so it'll be fast. If a chunk of the data is all on the same indexer, you are limited in search speed by that indexer's ability to get the data back.

Example: Imagine trying to get a 10GB file from a single host, vs 1GB files from 10 hosts. The bottleneck is reading from the host (not network), and as such, the 1GB from 10 hosts is going to me like 10x faster.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

There is no internal proxy setting for Splunk itself (although ES has a modular input for the Threatlists that allows for a proxy setting.) Instead you should be configuring your proxy at the OS level. Both *nix and Windows have this feature..

Here's One Example : http://www.cyberciti.biz/faq/linux-unix-set-proxy-environment-variable/

0 Karma

hiteshkanchan
Communicator

Has there been any progress on this issue in the recent times. I am trying to do a similar thing and not able to send the data through the proxy.

0 Karma

Yasaswy
Contributor

Hi ... sorry not much has changed on this front.

"Is there a setting somewhere that can tell the forwarder to connect to Splunk Cloud through a proxy"? I don't think so /No

As must have been explained in the earlier answers..... typically proxy connections are only for http requests. Your forwarder needs to connect over TCP on specific port to send the data... this may not be http. If the objective is to get the data into splunk cloud ... it will have be be designed and setup in collaboration with the network security and AWS teams. Eg: Setting up some standard servers as intermediate forwarders in your VPC and opening them up at the firewall might help.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...