Hi,  We recently migrated from a standalone Search Head to a clustered one. However, we are having some issue running some search commands. For example, this is a query that is not working on the new SH cluster sourcetype=dataA index=deptA | where critC > 25   On the old search head, this query runs fine and we see the results as expected. But on the SH cluster, this doesn't yield anything.  I have run the "sourcetype=dataA index=deptA" search query by itself, and they both see the same events. I am not sure why the search with (| where citC > 25) on the standalone SH would work and the cluster would not. Any help would be appreciated. Thank you   ... View more

Hi all, I would like to migrate our current cluster master to the a new server. Here's what I gather the process to do so. If someone can take a look and let me know if there's anything missing that'll be much appreciated. Thank you! Additionally, should I enable cluster maintenance mode on the old cluster master prior to the migration?  ======================================================================== ======================== Migrate the Cluster Master ==================== ======================================================================== - Stop the splunk service on both the old and new cluster master /opt/splunk/bin/splunk stop - On the old Cluster Master change encrypted passwords to clear text and save theses find /opt/splunk/etc -name '*.conf' -exec grep -inH '\$[0-9]\$' {} \; /opt/splunk/bin/splunk show-decrypted --value '$encryptedpassword' - - Copy files to the new Cluster Master scp -r /opt/splunk/var/run/splunk/cluster/remote-bundle/ new_splunkmaster:/opt/splunk/var/run/splunk/cluster/remote-bundle/ scp -r /opt/splunk/etc/master-apps/ new_splunkmaster:/opt/splunk/etc/ scp -r /opt/splunk/etc/system/local/server.conf new_splunkmaster:/opt/splunk/etc/system/local/ - Make sure the above decrypted the main 2 passwords below and replace them in the copied server.conf, in clear text, on the new Cluster Master until it is restarted when it will then encrypt. [general] sslPassword= [clustering] pass4SymmKey= - Start splunk on the new Cluster Master /opt/splunk/bin/splunk start - Point indexers to the new Cluster Master /opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://new_splunkmaster:8089 -replication_port 9887 -secret new_splunkmaster - Point the search heads to the new Cluster Master /opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://new_splunkmaster:8089 -secret new_splunkmaster ======================================================================== ======================== Migrate the License Manager ==================== ======================================================================== - Promote a license peer to be the manager: On the peer, navigate to Settings > Licensing. Click Switch to local manager. On the Change manager association page, choose Designate this Splunk instance as the manager license server. Click Save. Restart the Splunk Enterprise services. On the new license manager, install your licenses. See Install a license. Configure the license peers to use the new license manager: - On the peer (indexer / search heads / deployer), navigate to Settings > Licensing. Click Switch to local manager. Update the Manager license server URI to point at the new license manager. Click Save. Restart the Splunk Enterprise services. Demote the old license manager to be a peer: - On the old license manager, navigate to Settings > Licensing. Click Change to peer. Click Designate a different Splunk instance as the manager license server. Update the Manager license server URI to point at the new license manager. Click Save. Stop the Splunk Enterprise services. Using the CLI, delete any license files under $SPLUNK_HOME/etc/licenses/enterprise/. Start the Splunk Enterprise services.   ... View more