Hi soutamo, after a while I was able to do all the steps and now the old indexer/peers are removed from our environment. Everything works as expected. Thanks a lot for your help. ... View more

Hi soutamo, thanks for your answers and links to the docs. I have read the docs but I do not fully understood how it will work. Maybe you can help me again. 😉 This is the first time I have to do such an operation to our productive systems. Therefore I like to be sure that I do not do any mistake. This is what I understood from the docs: Stop new external data input Verify that all forwarders will send data only to the new peers. So that only internal data will be indexed for the moment. Is there a way to check if all incoming data are no longer use the "old" indexers? I checked the different menu's from Monitoring Console but could not find any usable report. So I tried searching with this SPL:   index=* source=* splunk_server="<old indexer1>" OR splunk_server="<old indexer2>" | stats count by host   Set "old" indexer peers in detention mode To stop indexing new internal and external data I have to switch the old indexers into detetion mode. Command to use from the command line of the "old" peer.   splunk edit cluster-config -auth <username>:<password> -manual_detention on   Initiate data rebalancing for all existing indexes by entering this command   splunk rebalance cluster-data -action start -searchable true Will the master automatically detect which indexes and data needs to be replicated?   Remove a peer from the master's list   splunk remove cluster-peers -peers <guid>,<guid>   I read I have to take the peer "down" or "gracefulshutdown" before I can remove the peer. Will it be enough to take the node offline? How can I check before removing the peer from the cluster that all buckets are replicated to the "new" indexers? Do you have any check search available? Hope I do not miss anything and the foreseen procedure is correct. As I said, this is the first time I have to do such a task. Many thanks. ... View more

@thaddeuslim, we found a new solution for this issue. We do not use snmp to get swift traps into splunk anymore. Sorry. ... View more

No error is been reported in any logfile. If I use index="test" host="testserver*" | fields _time, raw | sort 0 _time in the search field and search for the last 7 days. I got 10434 events. Then I choose a timeframe from the time picker and at the bottom of the page I didn't see any events, but the page show below the search field that I selected 394 events. Unfortunately I couldn't upload an example, otherwise you could see what I mean. ... View more

Just for information, we found a solution for our environment. It is pretty simple. We wrote a powershell script which is executed every startup of the provisioned servers. This script make the following changes: - remove the file "cloneprep" - add or replace the hostname from "inputs.conf" - add or replace the servername from "server.conf" - create or change the "instances.cfg" with a self created guid. We test this a couple of times and it looks like to work fine. The logfiles from the UF are looking good. No errors or warnings regarding the self created GUID. After all these changes from the configuration part, the script checks the successfull changes and startup the SplunkForwarder service on the server. For the moment our problem is solved with this script but if someone has another approach, please let me know. ... View more

@bcyates, thank you very much! You saved my day. Your suggested solution works fine. Just for information, I checked the latest version of splunk (7.0.2) there the authentication_roles.xml is the same as mine with version 7.0.0. So the issue isn't fixed by Splunk yet. ... View more

Could you manage the indexes.conf by the deployment server also for the cluster master? Do you have an example for me how to configure this? ... View more

So far, that sound good, but how do you set such a configuration up? In my case I have 2 indexers which are working in a cluster. My SH is also the ClusterMaster. I it could be even nicer but I do not have more machines yet. 😞 I try to configure the roles for differnet groups on the search head but unfortunately I couldn't see the created indexes in the web gui. Is there a way to make them visible there? By the way, the indexer cluster is working fine. Events are coming in and are indexed fine. ... View more

Does anybody else know how this work? Indexes are defined with the cluster, but the search head will not know about the indexes and therefore I'm not able to define some custom roles for these indexes. There must be a solution for that, but so far I do not find the answers in the documentation. Any help would be really nice and welcome. Thanks. ... View more

As i said, the indexes.conf file is located on the indexers in /opt/splunk/etc/slave-apps/_cluster/local. The incomming events are correct indexed and are searchable over the indexers as well as on the SH. So this configuration seems to be okay. It is only for the role setup. There I'm not able to select the different indexes for the roles. ... View more

Hi, thanks for the link and your answer. I read the part of the documentation this morning and decided to create a dedicated app for the authorization. Unfortunately I do not have as much knowledge about the indexer cluster and a dedicated SH. Therefore I have to ask again. You wrote I should be able to manage the roles by the Web interface. But if I login to the SH with the admin account, I cannot see any custom indexes. I only see the defaults. Just for your information. All the custom indexes are created with an indexes.conf file on the SH which is also the Cluster Master in the directory /opt/splunk/etc/master-apps/_cluster/local. I pushed this information to the indexers by checking and pushing it with the web interface from the SH / CM. My new authorization app is located on /opt/splunk/etc/shcluster/apps. Because I checked the configuration in the webgui and both indexers are defined as search peers. How can I deploy this app now to the search peers? Is it correct that with this configuration, I'm not able anymore to use the webgui for the management of the roles? Many thanks. ... View more

On the old DS I see the entries from the 6.2.0 Forwarders, but the "Phone Home" field is "4 days ago". Thats the time when I changed the deployment server configuration in the deploymentclient.conf on the Forwarders. But I will try to upgrade now one of the 6.2.0 Forwarders to the 7.0.0 version and will see what happens then. [edit] I did an upgrade from 6.2.0 to 7.0.0 on one of the Forwarders who wont communicate with the DS. Since the version was changed the Forwarder communicates with the DS and I'm able to deploy an custom app. So it looks like the communication between 6.2.0 Forwarders and 7.0.0 DS is not possible/supported. ... View more

Yes, the old 6.2.0 Deployment Server is still up and running. Could this be the issue? I will upgrade one Forwarder to see what happens then. I will let you know. ... View more

Hi Giuseppe, thanks for you answer. You are absolutly right. It is not recommended to use Search Head and Deployment Server on the same server. But for my test it should be okay. Unfortunately I cannot see any 6.2.0 forwarders under the deployment server (forwarder management). I only see there the forwarders which has installed the 7.0.0 version. On the old (6.2.0) forwarders I found such messages into the splunkd.log "INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected" I don't know why it tells me "not_connected". ... View more