Basic Distributed Deployment Install/Setup Steps

[marycordova]

Is there a basic cheatsheet for setting up a new small scale distributed deployment?

[marycordova · Accepted solution]

• Here are the basic steps for setting up a new deployment. 
• Build your Deployment Server first through step 8.  You can use the WebUI on the Deployment server to upload your license file. 
• Build your indexers, heavy forwarders, & search head with the same steps adding items 9-13. 
• Be sure you also configure your Deployment server with your indexes, SSL, and forward the internal logs to your indexers. 
• These steps were done on Ubuntu so commands may vary slightly on other *nix flavors.  You may have other ways (systemd) to do some of the *nix admin such as the limits and THP.

• login via cli & elevate to root
• increase system limits

vi /etc/security/limits.conf


* hard nofile 64000 
* hard nproc 16000 
* hard fsize -1

• disable THP
  • https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/ System V Init (service)

vi /etc/init.d/disable-transparent-hugepages


#!/bin/bash
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: splunk
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
### END INIT INFO

echo 'never' | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null
echo 'never' | tee /sys/kernel/mm/transparent_hugepage/defrag > /dev/null


chmod 755 /etc/init.d/disable-transparent-hugepages
/etc/init.d/disable-transparent-hugepages start
update-rc.d disable-transparent-hugepages defaults

• reboot, login, & elevate to root
• create the splunk user

useradd -m splunk

• install splunk & configure to run as non-root "splunk" user at system boot

cd /opt
wget -O splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz "https://download.splunk.com/products/splunk/releases/9.0.4.1/linux/splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz"
tar zxfv splunk-9.0.4.1-419ad9369127-Linux-x86_64.tgz
chown -R splunk:splunk /opt/splunk
/opt/splunk/bin/splunk enable boot-start -user splunk

• reboot, login, & elevate t root
• verify configs
  • https://docs.splunk.com/Documentation/Splunk/9.0.4/RESTREF/RESTintrospect

ulimit -a
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
ps -ef | grep splunk

• switch to splunk user

su - splunk

• add deploymentclient.conf

vi /opt/splunk/etc/system/local/deploymentclient.conf


[deployment-client]
[target-broker:deploymentServer]
targetUri = https://deploymentserver.yourdomain.com:8089

• add the Splunk license, restart Splunk, & check licensing

/opt/splunk/bin/splunk edit licenser-localpeer -manager_uri 'https://deploymentserver.yourdomain.com:8089'
/opt/splunk/bin/splunk restart
/opt/splunk/bin/splunk list licenser-localpeer

• add to appropriate Deployment Server Class(es) & deploy appropriate apps
  • enable (SH, HF, DS) or disable (indexers) the WebUI
  • enable cooked Splunk port 9997 inputs on the indexers
  • forward _* internal logs to the indexers
  • define indexes
  • enable SSL
    • WebUI, 8089 mgmt, 9997 cooked input
• enable email (SMTP) for SearchHead only via the SH WebUI 

[marycordova]

I'll try to add a little more detail here when I get a chance:

• add to appropriate Deployment Server Class(es) & deploy appropriate apps
  • enable (SH, HF, DS) or disable (indexers) the WebUI
  • enable cooked Splunk port 9997 inputs on the indexers
  • forward _* internal logs to the indexers
  • define indexes
  • enable SSL
    • WebUI, 8089 mgmt, 9997 cooked input

[inventsekar]

Hi @marycordova ... 

As per my knowledge there are no cheetsheet kind of documents.. 

but the Splunk documentation is very detailed and it will give us all the info required. 

Please install SH, indexers, as per this document:

https://docs.splunk.com/Documentation/Splunk/9.0.4/Installation/InstallonLinux

then you can configure the indexer cluster and search head cluster as per this document:

https://docs.splunk.com/Documentation/Splunk/9.0.4/Deploy/SHCwithindexers

hope this helps.. thanks.