I realize this response comes in a bit late in the game for most on this thread, but hopefully it will be useful to someone.  KV Store is not used on indexers. In fact, it can be disabled on any Splunk Enterprise instance that is not running as a search head (see caveats below). This means includes Indexers, Deployment Servers, Cluster Managers, License Managers and Search Head Cluster Deployers, Heavy Forwarders (see exceptions below). Instances that SHOULD run KV Store include: Search Heads Monitoring Console (if you want access to its KV Store dashboards-- see caveats below) Heavy Forwarders, if used as search heads or are running DB Connect app (or, not likely but possible, any custom app that specifically relies on the KV Store)  Lastly, any Splunk Enterprise instance in your deployment which requires token-based authentication CAVEATS: If your Monitoring Console machine also operates another Splunk role such as Deployment Server or License Manager, keep the KV Store running on that instance. As of this writing, the Monitoring Console appears to use the KV Store only for KV Store-specific dashboards. If you don't care about those dashboards and the KV Store is giving you real headaches on that instance, disabling it will cause the dashboards which rely on KV Store to stop working, but all other Monitoring Console dashboards will function as expected. Universal Forwarders are not shipped with the KV Store so nothing to worry about there ... View more

A standalone Deployment Server that is not functioning with any other server roles should be able to handle up to 25,000 forwarders. In the past, customers with large deployments have set up Deployment Servers behind a load balancer and kept apps sync'd between them using tools such as Puppet or Ansible. Since 9.2.0, Splunk Deployment Servers have been architected to work as a "cluster" behind a load balancer and to keep apps and client status sync'd between them via a shared network directory. This allows any number of forwarders to be managed. For example, for an environment capturing data from 100,000 forwarders, a cluster of at least 4 Deployment Servers would be a good place to start. ... View more

Since Splunk Enterprise 9.2.0, Splunk has introduced "Deployment Server Scaling", which involves setting Deployment Servers behind a load balancer (or use DNS mapping) and granting all access to a single network share. Each DS uses the share path to update and share app configurations and post log files. This allows the DS' to keep apps, client lists and client status in sync between them. While Splunk documentation mentions 50 clients, this is only in reference to ensuring the DS is on its own server, not sharing functionality with any other Splunk instance such as search head, indexer, Monitoring Console, License Manger, etc. A Deployment Server can actually handle up to 25,000 clients, if granted enough system and network resources to manage the load. With Deployment Server scaling, the number of forwarders that can be managed multiplies with each Deployment Server added to the "cluster". Two can manage up to 50,000 clients, three can manage up to 75,000, etc. All Deployment Servers in a cluster share all apps and all clients. DS Scaling is also referred to as "clustering", though it works nothing like indexer or search head clusters-- the different DS's don't communicate with one another directly or formally form a "cluster".  This allows very large environments to manage a multitude of forwarders. Too many forwarders? Add another Deployment Server. Here are a few links: Splunk Documentation: Implement a Deployment Server Cluster Splunk Documentation: Estimate Deployment Server Performance Deployment Server section of this Splunk Lantern article: Scaling your Splunk Deployment, which consolidates relevant Splunk documentation Splunk Community Article: Deployment Server Scalability Best Practices "Discovered Intelligence" blog article on setting up a Splunk Deployment Server cluster. I have not (yet) tested their suggestions but this is a great place to start for a quick overview of what's needed. Deployment Servers are on track for significant improvements in the near future as well, with the goal of reducing/eliminating the need for 3rd party tools such as Puppet or Ansible for those who wish to manage everything within Splunk itself. ... View more

Thank you for that, @dhatch ! This feature should be defaulted to "true" in $SPLUNK_HOME/etc/system/default/server.conf, starting with Splunk 9.1.x. In previous versions, according to the web.conf.spec file, this attribute is not set at all. Whenever making changes to .conf files in Splunk, please edit files from $SPLUNK_HOME/etc/system/local. If you do not yet have a web.conf file in that path when making such a change, create it, then include only the stanza(s) and attribute(s) you wish to modify. Settings configured via $SPLUNK_HOME/etc/system/local take precedence over all other configuration files in all Splunk instances EXCEPT for clustered indexers. In the case of clustered indexers, the apps deployed to indexers by the Cluster Manager take precedence even over $SPLUNK_HOME/etc/system/local.  See: Splunk documentation on precedence This useful community article   ... View more

@ankitarath2011 ,  Agreed, generally it is recommended to instead blacklist items in your search head apps that don't need to be sent to the indexers, as this will keep your bundle sizes down. A few additional suggestions: The Admins Little Helper for Splunk app can help identify bundle contents (and computed/expected contents) For more information about bundle size, refer to https://docs.splunk.com/Documentation/Splunk/latest/Admin/Distsearchconf   maxBundleSize = <int> * The maximum size (in MB) of the bundle for which replication can occur. If the bundle is larger than this bundle replication will not occur and an error message will be logged. * Defaults to: 2048 (2GB)​   and: Large lookup caused the bundle replication to fail. What are my options Alternatively, you may find large lookups and configure limits.conf accordingly: To find large lookup files check bundles on the indexer in $SPLUNK_HOME/var/run/searchpeers/. Copy one of the bundles from $SPLUNK_HOME/var/run/searchpeers/ over to a tmp dir and run:   tar xvf <file>.bundle find -type f -exec du -h {} \; | grep .csv   Set max_memtable_bytes larger than the largest lookup in limits.conf:   [lookup] max_memtable_bytes = 2*<size of the largest lookup> example: On all indexers in limits.conf set: (for unclustered indexers: $SPLUNK_HOME/etc/system/local/limits.conf; for clustered indexers, use the _cluster app on the Cluster Manager or if you distribute indexes.conf and other indexer settings in a custom app, place it in there) [lookup] max_memtable_bytes = 135000000 #equals to 135MB, where the largest lookup file was 120MB   ... View more

@ankitarath2011 , Apologies for the delay, as I have been out of the office. The issue you are reporting is very different than what is discussed on the current post, and needs a new thread. Could you rephrase your full question in a new post, and tag me in it? I tried to start a new one for you that we could continue on but I'm not certain the full context of your issue and question. Regarding increasing maxBundleSize, it is normally a better practice to manage bundle sizes using the [replicationWhitelist] or [replicationBlacklist] stanzas in distsearch.conf.  Raising bundle size limits or raising bundle replication timeouts can cause bundles to take longer to reach your indexers. By default, Search Heads use knowledge bundles to send nearly the entire contents of all of their apps to the indexers. If an app contains large binaries that do not need to be shared with the indexers, reduce the size of the bundle by whitelisting or blacklisting particular files or types of files. See: Splunk Documentation: Limit the knowledge bundle size    Also as an aside... in the event this might be helpful, Admins Little Helper for Splunk can be used to view bundle contents (and computed/expected contents). I will be out this coming week also but will check periodically for your new post. I will not respond on this thread. Thank you, ... View more

@ankitarath2011 , the issue discussed in this post occurs in Splunk 9.0.0 and 9.0.1. It does not occur in 8.x versions nor in versions from 9.0.2 onward. How is your environment currently configured? See https://docs.splunk.com/Documentation/Splunk/8.2.3/DistSearch/PropagateSHCconfigurationchanges.  What deployer push mode are you using, and where are you setting this value in app.conf? Within an app on the search heads, or in $SPLUNK_HOME/etc/system/local/app.conf on the Deployer? ... View more

Actually, the correction was made to 9.0.2. See my comment above... which also provides a workaround for those running 9.0.0 or 9.0.1. ... View more

Are you using a custom certificate, and is it secured with a password? This error suggests an incorrect password, or possibly the password uses special characters and mongod doesn't like them. I found a case where someone using mongod and logging in from the command line had to escape out the special characters in their password. I don't see how that would be an issue with Splunk's implementation o mongod, but might be a good thing to test. In the meantime, I will continue researching as I am working with another Splunk user encountering the same message. ... View more

Your issue may have to do with changes to deployer_lookups_push_mode setting in app.conf. See explanation in 9.0.2 README: In 9.0, a change was made to the behavior of the default preserve_lookups value for the deployer_lookups_push_mode setting in app.conf. This change fixed a behavior issue that caused preserve_lookups to not conform to its documented and intended behavior. Rather, prior to 9.0.0, the preserve_lookups value conformed instead to the documented behavior of the always_preserve value. However, although the 9.0.0 change fixed the behavior of the preserve_lookups value, the change also led to performance degradation when using that value because of additional processing needed to attain the intended result. In addition, the fix changed the default behavior of the deployer_lookups_push_mode setting, which introduced an additional problem, since some users had come to expect and rely on the pre-9.0.0 behavior of the default preserve_lookups value, buggy though it was, To counteract the resultant performance degradation and change to expected default behavior introduced by the 9.0.0 change to the default value, in 9.0.2, the default value for the deployer_lookups_push_mode setting was changed to always_preserve. This change to the default causes the default behavior of the setting to conform to the unfixed behavior of the preserve_lookups value prior to the change in 9.0.0 Documentation has also been upgraded for default behavior of the deployer_lookups_push_mode setting in app.conf for 9.0.2. See: https://docs.splunk.com/Documentation/Splunk/9.0.2/DistSearch/PropagateSHCconfigurationchanges#Preserve_lookup_files_across_app_upgrades If this is your issue, options may be: Upgrade to 9.0.2; or, Configure your 9.0.0 or 9.0.1 instance like 9.0.2, so it will function like pre-9.0 releases. As such: Update global setting on deployer to always_preserve in $SPLUNK_HOME/etc/system/local/apps.conf: [shclustering] deployer_lookups_push_mode = always_preserve  Update specific apps, if needed, to the preferred setting for that specific app: $SPLUNK_HOME/etc/shcluster/apps/<app>/local/app.conf [shclustering] deployer_lookups_push_mode = preserve_lookups I hope this helps! ... View more

Quite some time has passed since this question was posted, but a similar question came up today... Most likely, the processes need to remain separate so the CM can cleanly update journal.gz on all decommissioned site indexers to the site they are moving to, and to allow the CM to meet SF/RF on the remaining site before changing it to a single-site cluster. This assumes you set the remaining site's SF/RF to be identical to what it would be when it becomes single-site. If this process CAN be combined, I suspect it is possible if site_mappings attribute in CM's server.conf was not removed, but instead updated to <decommissioned_site_id>:<remaining_site_id>. site_mappings = site2:site1 When the CM comes back up, it will very quickly update journal.gz on all site2 indexers so the buckets are assigned to site1. Assuming of course the cluster was placed into Maintenance Mode during this process, as doing so will keep the CM from initiating unnecessary fix-up tasks.  ... View more

Yes, permissions can get pretty messy. This also happens if Splunk is normally run by the Splunk user but is then started from root (usually using sudo). ... View more

Agreed! I had reproduced the issue again on another instance and wanted to document the steps to changing ownership. It's a little different depending on whether the original search was shared or not. Here's a recap of my findings: Alerts on Dev/Test licenses do work if the account for the instance IF: The login account is named 'admin' (the Dev/Test license only allows one user account, and it must be named 'admin'); AND In Advance Settings for the alert, the server is changed from 'localhost' to your organization's email server (for me, mail.splunk.com). To change the name of the login account to 'admin', update the passwd file to reference ":admin:" at the start of it instead of your original username (example: ":choppedliver:"), then restart Splunk. To copy the search history and saved searches / alerts over to the admin account: copy the history and local directories from $SPLUNK_HOME/etc/users/<originalusername> over to $SPLUNK_HOME/etc/users/admin. You can delete them from the <originalusername> folder also if you wish. NOTE: this instruction assumes your admin account is fresh and doesn't already have a history or local directory. If you have been using the admin account, decide which search history you want to keep. Combining them doesn't seem to work. If you already created saved searches for the admin account, copy the contents of the local/savedsearches.conf file from your original user over to the savedsearches.conf file currently existing for your admin account. For shared alerts, update the local.meta file to assign shared alerts from <originalusername> to admin ($SPLUNK_HOME/etc/apps/search/metadata/local.meta) Once again, restart Splunk for these changes to go into effect. ... View more

Let me see what I can find out about that. Before updating the passwd file I had tried copying things over and was not getting any changes/results from doing so. ... View more

Glad to hear this solution helped, @dkozinn ! This thread seemed dated, but it sent me in the right direction for troubleshooting, so figured I'd post my findings anyway. It appears to occur across Splunk versions also. I also noticed the disappearing search history, and it did seem to come back when the account was renamed to 'admin', at least up to the point where the issue started. I had tested both using searches with "sendemail" and setting up scheduled alerts. The scheduled alerts also came back online once the account was renamed 'admin'. ... View more