Can You Help Me Understand The Environment I Inherited

[paimonsoror]

So take this with some warning.... its a bit of a mess.

This is our nonprod environment, and the goal was to move our infrastructure from a private cloud that was severely underpowered, to a virtual environment that has been appropriately scaled to Splunk recommendations.

Old Environment Servers will be referred to as cloudX
New Environment Servers will be referred to as virtualX

The old cluster was made up of the following:
1 Deployer, 3 SHs, 1 Cluster Master, 5 Idx

The new cluster is made of the same configuration

We currently have all users pointing to the new environment. And here is where I get lost. When I perform a search, I actually see the search going out to all 10 IDx. So I thought, ok, maybe the setup was made to have the 5 old IDX as read only, while the 5 new IDX would consume all the new data. Eventually allowing us to fade out the old servers. This however doesn't seem to be the case, as running a search from just this morning sees the following from the Inspector:

10.50   dispatch.stream.remote.cloud3   67  -   8,365,056
1.47    dispatch.stream.remote.cloud1   59  -   1,934,223
0.18    dispatch.stream.remote.cloud0   10  -   193,095
0.00    dispatch.stream.remote.virtual0 4   -   18,650
0.00    dispatch.stream.remote.virtual1 4   -   18,666
0.00    dispatch.stream.remote.virtual2 3   -   14,023
0.00    dispatch.stream.remote.virtual3 1   -   4,737
0.00    dispatch.stream.remote.virtual4 1   -   4,738

So I tried digging a bit to see where these old servers are still being used in my cluster...

I opened the virtualClusterMaster and took a look, and i see the new (virtual) indexers

virtual0     Yes    Up  12
virtual1     Yes    Up  113
virtual2     Yes    Up  14
virtual3     Yes    Up  84
virtual4     Yes    Up  112

Interestingly enough, looking at the virtualDeployer i see the old (cloud) cluster listed, and not the new (virtual) one

Would this be enough information to help determine what might be going on here? I can understand that data is probably still going to the old indexers because of the forwarder configurations, but what I am not understanding is how the cluster knows to look at those old (cloud) indexes.

[lguinn2 · Accepted solution]

I would take a look at the configuration of the search heads. A search head can participate in multiple clusters, so I suspect that your search heads are searching both the old and the new.

You probably should make sure that all the forwarders are sending data to the new cluster ASAP. Or else you will never be able to power-off the old cluster.

You might also look at different timeranges for the searches - you didn't say what the timerange was for the inspector info that you shared. You might find that a search running over the last 24 hours has a very different profile than a search over the last 30 days. (At least I hope.)

[paimonsoror]

Thats a great point. I certainly need to reset those forwarders to send data to the new cluster soon. Thanks for this information by the way. I took a look at the cluster master on the old server, and I see some references to the new search heads....so that must be the key right there.

[lguinn2]

Look at the .../etc/system/loca/server.conf on the search heads, too

[paimonsoror]

THERE IT IS! ... you are the best, thank you!