So take this with some warning.... its a bit of a mess.
This is our nonprod environment, and the goal was to move our infrastructure from a private cloud that was severely underpowered, to a virtual environment that has been appropriately scaled to Splunk recommendations.
Old Environment Servers will be referred to as cloudX
New Environment Servers will be referred to as virtualX
The old cluster was made up of the following:
1 Deployer, 3 SHs, 1 Cluster Master, 5 Idx
The new cluster is made of the same configuration
We currently have all users pointing to the new environment. And here is where I get lost. When I perform a search, I actually see the search going out to all 10 IDx. So I thought, ok, maybe the setup was made to have the 5 old IDX as read only, while the 5 new IDX would consume all the new data. Eventually allowing us to fade out the old servers. This however doesn't seem to be the case, as running a search from just this morning sees the following from the Inspector:
10.50 dispatch.stream.remote.cloud3 67 - 8,365,056
1.47 dispatch.stream.remote.cloud1 59 - 1,934,223
0.18 dispatch.stream.remote.cloud0 10 - 193,095
0.00 dispatch.stream.remote.virtual0 4 - 18,650
0.00 dispatch.stream.remote.virtual1 4 - 18,666
0.00 dispatch.stream.remote.virtual2 3 - 14,023
0.00 dispatch.stream.remote.virtual3 1 - 4,737
0.00 dispatch.stream.remote.virtual4 1 - 4,738
So I tried digging a bit to see where these old servers are still being used in my cluster...
I opened the virtualClusterMaster and took a look, and i see the new (virtual) indexers
virtual0 Yes Up 12
virtual1 Yes Up 113
virtual2 Yes Up 14
virtual3 Yes Up 84
virtual4 Yes Up 112
Interestingly enough, looking at the virtualDeployer i see the old (cloud) cluster listed, and not the new (virtual) one
Would this be enough information to help determine what might be going on here? I can understand that data is probably still going to the old indexers because of the forwarder configurations, but what I am not understanding is how the cluster knows to look at those old (cloud) indexes.
... View more