The files are available here: https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 THe FAQ sent by Splunk for expiry on July 21, 2016: Dear Splunker, Enclosed are remediation steps for the expiration of default certs shipped with Splunk 6.2 and earlier. Please let us know if these steps were helpful. SUMMARY OF THE ISSUE In a nutshell, default certificates shipped with 6.2 and earlier versions of Splunk have expired and will affect communications between Splunk components: HTTPS - between browser and Splunk will fail if the following is set: webconf: EnableSplunkWebSSL, confirm default certs are targeted in privKeyPath and caCertPath. See workflows provided. You will want to swap out the default certs (download at link below) or run the s-renewalcerts scripts. UF and IDX - This will fail IF SSL is set for the type of connection between the forwarder and indexers. FYI, dire error messages will be written to logs even if communications between fwd-idx are working just fine. You will want to swap out the default certs (download at link below) or run the s-renewalcerts scripts. Most Everything else (Deployment Server - Deployment Client, License Master to License Slave, Distributed Search Server-Client, Search Head to Search Peers) will be able to be turned off with the following flags: Forwarder (client) sslVerifyServerCert = false Indexer (server) requireClientCert = false First Response FAQ Q: Am I using SSL to encrypt between the forwarder and indexer? A: Run this search on the indexer and a true result will confirm use of SSL to encrypt communications between the forwarders and indexers indicated. index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl If the results are false, forwarder - index traffic will not fail. Q: How do I check the expiration for all certs on the customer's system? A: A quick way to check all expiration dates for all certs under $SPLUNK_HOME/etc: $SPLUNK_HOME/etc/$ find ./ -name '*.pem' -exec openssl x509 -in '{}' -noout -enddate \; Q: What are my options for immediate remediation? A: Run the script provided in the advisory. The advisory: https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-lig ht-and-hunk-pre-63.html Or, manually swap out the expired default certs with default certs provided here: https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 See steps below. Q: What if I can't get the script to work? Do we have manual steps? A: Yes. Here they are: 1) Stop Splunk 2) Back up (note permissions/ownership, very important!!) and then remove: · ./etc/auth/ca.pem · ./etc/auth/cacert.pem · ./etc/auth/server.pem 3) Copy over the new versions of the following from https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 · ./etc/auth/ca.pem · ./etc/auth/cacert.pem · Do not copy over server.pem 4) Confirm permissions and ownership of 3). 5) Restart Splunk. A new server.pem is generated. You're done. Q: How can I tell which SSL parameters have been set in my configuration files? A: We recommend using btool along with grep splunk btool server list --debug | grep blargh Check for: [sslConfig] enableSplunkdSSL = true caCertFile = cacert.pem caPath = $SPLUNK_HOME/etc/auth sslVerifyServerCert = true splunk btool outputs list --debug [tcpout:splunkssl] sslCertPath = $SPLUNK_HOME/etc/auth/server.pems RootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem splunk btool inputs list --debug [SSL] rootCA = $SPLUNK_HOME/etc/auth/cacert.pem serverCert = $SPLUNK_HOME/etc/auth/server.pem [splunktcp-ssl:] Additional Resources The advisory: https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-light-and-hunk-pre-63.html Download the script: http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip Splunk Wikis: http://wiki.splunk.com/Community:SplunkWeb_SSL_DefaultCerts http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts Thank you, The Global Customer Support Team ... View more

For upgrades from an earlier version to 6.3.x, please remove existing copies of ca.pem and cacert.pem before upgrade. Steps for Linux: Stop Splunk Remove $SPLUNK_HOME/etc/auth/ca.pem Remove $SPLUNK_HOME/etc/auth/cacert.pem Upgrade procedure as usual (untar tarball over Splunk home directory) Start Splunk (this will generate a new ca.pem and cacert.pem files) Hope this helped anyone wondering why their upgrade did not work to change the expiration dates on their default certs. Just in Case: If the customer generated certs and gave them the names used by Splunk (ca.pem, cacert.pem), this answer does not apply. This answer only applies to default certs provided out of the box by Splunk. ... View more

For context, this answer refers to VMWare App 3.1.3 on Splunk 6.x. In the Splunk App for VMware tsidx files are stored in namespaces on the search head. You can manage the size of the tsidx namespace files using a retention policy. A retention policy specifies a limit for the size of the tsidx namespaces created and the length of time that namespaces are retained in your environment. Background info on the recommendations below: http://docs.splunk.com/Documentation/NetApp/2.0.1/DeployNetapp/Considerationswhenusingtsidxnamespaces http://docs.splunk.com/Documentation/VMW/3.1.1/Configuration/Managenamespacefilesize http://docs.splunk.com/Documentation/NetApp/2.0.2/DeployNetapp/Considerationswhenusingtsidxnamespaces Adjust a maximum size limit or maximum retention time by copying /etc/apps/SA-Utils/default/tsidx_retention.conf to the local directory. The default retention time is 2,184 days, and the maximum size of an index defaults to 500G: [default] maxTotalDataSizeMB = 500000 retentionTimePeriodInSecs = 188697600 Copy and move to ../local/tsidx_retention.conf on the search head to enable changes. The following changes adjust max size of the index to 50G, and changes the retention time to 7 days: [default] maxTotalDataSizeMB = 50000 retentionTimePeriodInSecs = 604800 The maximum size of all buckets for an index is maxTotalDataSizeMB . Also, there is a scripted input that runs once a day at 3:00AM that cleans up the tsidx files. The input is defined in /etc/apps/SA-Utils/default/inputs.conf. The default is probably fine, but scheduling can be adjusted or the script can be run manually if desired. [script://./bin/tsidx_clean_up.py] disabled = false passAuth = splunk-system-user # Once per day at 3 AM interval = * 3 * * * index = _internal sourcetype = tsidx:clean_up ... View more

I looked everywhere and couldn't find this, so please upvote if this helped: If you want to set Splunk to start up with a non-default port, non-interactively, before starting up for first time: Copy the default web.conf from $SPLUNKHOME/etc/system/default/web.conf to $SPLUNK_HOME/etc/system/local/web.conf. (it will not exist in /local until created) $ cp $SPLUNKHOME/etc/system/default/web.conf $SPLUNKHOME/etc/system/local/web.conf More on file precedence here: Remove lines unchanged. Just use the example below if you have no other changes. To change the splunk web server port from the default value of 8000 to 12300, and the default splunk management port from 8089 to 12389, see example below. [default] [settings] httpport = 12300 mgmtHostPort = 127.0.0.1:12389 Save the file. Start splunk. $ SPLUNK_HOME/bin/splunk start Voila! It should confirm ports opened during the start up process: > Checking prerequisites... > Checking http port [12300]: open > Checking mgmt port [12389]: open ... View more