@delgendy - You have to step back and apply root principles to the problem set first. If all the data you are going to have is 95th percentile and max duration, then there's no point in using the machine learning stuff, however cool it is. If you timecharted that, you might be able to see a trend. If you clustered it, you might be able to detect groupings of indexes. But machine learning is inappropriate to the data.
If you want to predict response times, and you want to use machine learning, and you want to take EVERYTHING into account, then you can start by thinking of all the things that could affect response times (time of day, day of week, cpu load, index, type of API call, etc) and figure out how to collect and clean and prep and correlate all that data.
Then, having collected and prepped a decently-sized mess of data, the machine learning may be able to explain something interesting to you. (For instance, you may find that a particular call at a particular time of weekday predicts a slow response, and from investigating that you may find that there is a competing process scheduled at that point that needs to be optimized.)
... View more