Yes, remove the files that are no longer being updated, or blacklist files that you are not actually interested in.
The monitor input was designed to pick up data as it is added to a file, so simply enabling it for thousands of static files is actually using it in the wrong way, as it will always go back and check files to see if they have been updated.
Using this method for a first time load is fine, as long as you update your inputs once that initial data-load is complete. Leaving it in place for a few hundred files is also fine, as Splunk can check this many files relatively quickly. As you increase the number of files being monitored however, you are slowing down how quickly new data is picked up.
I suspect that you are actually monitoring more files than you think, or perhaps you are using a NFS mount - network latency is also an important factor
