Do you have any guidance about how to revitalize a Splunk Enterprise/Splunk Cloud deployment that's stuck on autopilot?

[jmulcaster_splu]

I have a Splunk Enterprise/Splunk Cloud deployment that's been on autopilot for a while. We've been adding data sources and use cases, but I think there's a lot more we can get out of Splunk, and I'm not sure where to start. Do you have any guidance for how I can evaluate and revitalize my Splunk deployment?

[jmulcaster_splu · Accepted solution]

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

There is a lot you can do, and hopefully this post can get you started. The suggestions here apply to both Splunk Enterprise and Splunk Cloud–we included links to Splunk Cloud documentation, but the same principles apply to Splunk Enterprise.

Review and revamp your Splunk Cloud deployment

First, evaluate your administrative overhead and consider ways to simplify. For example, you may find ways to streamline data consolidation, load balancing, or data routing.

Think about the type of data that Splunk Cloud accepts and if using forwarders or Splunk apps to get data in from remote systems can strengthen or expand your Splunk usage. Use forwarders to get data in, or use Splunk apps to get data in and install a universal forwarder to ingest remote data from other systems into Splunk Cloud.

After you configure your data sources per the best practices for defining source types, review your forwarder deployment topologies and look for ways to optimize your data flow. Apply best practices for creating indexes, options for getting data into Splunk Cloud, and install a universal forwarder to ingest remote data from other systems into Splunk. If you need a refresher, watch the video Getting Data in to Splunk Cloud to see how to use the Universal Forwarder to forward data to the Splunk Cloud service.

The Inherit a Splunk Enterprise Deployment manual provides exhaustive steps for re-approaching your Splunk deployment to free it from a stale status quo. This includes, but is not limited to, topics about updating the architecture diagram, monitoring system health, and investigating problems with knowledge objects.

Scale efficiently with a deployment server

If you have many forwarders, a deployment server (DS) is often the best available software configuration management solution. Use a DS to manage to manage all Splunk instances within your infrastructures at scale.

A deployment server provides a single interface to distribute apps and manage configuration and content updates to existing Splunk installations on your network. A deployment server can filter based on hostname, IP address, or machine type. Review the deployment server architecture and efficiently distribute resources in your environment. After you set up the deployment server, configure deployment clients to receive data from the deployment server, and use the forwarder management interface to simplify forwarder and app configuration management.

[sloshburch]

Thanks @woodcock! I just added a paragraph to incorporate your point about the documentation available. Karma points are on their way!

[woodcock]

This blog post points to a good documentation page that will get you started:
https://www.splunk.com/blog/2017/05/02/inheriting-a-splunk-enterprise-deployment.html
Once you get your bearings on everything and put together an architecture diagram, review the error logs in index=_* and fix whatever you can figure out. At that point I would bring in some PS for a health-check/tune-up; many companies, including mine, provide this service.