Can I use forwarders to scale my Splunk Cloud deployment?

[adukes_splunk]

Where can I find more information about using forwarders to manage my Splunk Cloud deployment?

[adukes_splunk · Accepted solution]

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

Scale easily with deployment servers and forwarders

To optimize, and effectively reduce administration overhead, you can use a forwarder instance as a deployment server (DS) to deploy apps in your local network.

How a deployment server and forwarders help you scale

Splunk Cloud does not provide a deployment server. However, you can use forwarders to mimic this DS behavior in your Splunk Cloud environment and use it to distribute apps from your Splunk Cloud stack and deployment client.
Read about the types of forwarders to get an overview of how forwarders work and see a comparison of their features and capabilities. Today we’ll outline a configuration using a heavy forwarder and a universal forwarder that you can scale based on your needs. The configuration is based on Splunk’s Professional Services Base Configurations toolset.

Things to know

A deployment server is a great way to distribute apps on your network. Unfortunately, you cannot use a deployment server to manage index clusters or search head clusters, or upgrade installations of Splunk. You can use a dedicated heavy forwarder instance as a deployment server by placing it on the network with open firewalls for the Splunk Management Port to the DS host or you can deploy multiple DSs.
A DS can filter based on hostname, IP address, or machine type. So, we have a few options for deploying to all our clients.

Avoid using automation such as Puppet, Chef, or Ansible in conjunction with DS because it can cause .configs to disappear and break. Do not test your serverclasses.conf because it changes in a DEV environment.

• Receiver: A Splunk Enterprise instance that receives data from a forwarder and can be an indexer or a forwarder.
• Heavy Forwarder: A type of forwarder with a smaller footprint than a Splunk Enterprise indexer but retains most of the capabilities of an indexer. It cannot perform distributed searches. It’s best used to route event-based data.
• Universal Forwarder: A dedicated, streamlined version of Splunk Enterprise that contains only the essential components needed to forward data. It is usually the best way to forward data to indexers. Learn more about the universal forwarder in the Universal Forwarder Manual.
• Deployment: A set of distributed Splunk instances, working together.
• Deployment server: A Splunk instance that acts as a centralized configuration manager, grouping together and collectively managing any number of Splunk instances.
• Deployment client: A Splunk instance that is remotely configured by a deployment server.
• Server class: A group of deployment clients that facilitate the management of a set of deployment clients as a single unit.
• Deployment app: A unit of content deployed by the deployment server to a group of deployment clients. Deployment apps can be fully developed apps, such as those available in Splunkbase, or they can be a simple group of configurations.

Things to do

Some of the following are to Splunk Enterprise manuals. However, they are applicable to Splunk Cloud when following the general procedure in the Things to know section:

• Plan your deployment. Plan your deployment and consider some useful topologies that you can create with forwarders.
• Manage the deployment server. Manage the deployment server to provision deployment server resources and estimate how long it will take to download your apps to a set of clients.
• Set up a client. Configure deployment clients to receive data from the deployment server. Use the forwarder management interface to manage the update process across all Splunk instances.
• Deploy an app to your clients. Create a server class to map a group of deployment clients to one or more deployment apps to update the distribute configuration.
• Learn how to set up Universal Forwarder. Watch the Getting Data into Splunk Cloud video to see how to use the Universal Forwarder app to forward data to the Splunk Cloud service.

[djl]

I appreciate the wave of best practices answers that you have been posting recently.

When we were setting up our Splunk Cloud deployment the amount of documentation that was available regarding an intermediate forwarding tier consisting of Universal Forwarders was a bit lacking. Would it be possible to pull together best practices for that use case?

[sloshburch]

Thanks @djl - That's a topic in our backlog. I'll make a note of your interest if that helps increase it's priority for us. Thanks for being a follower!