It's almost like asking when to buy a new car or change your hair style there are a lot of factors to consider. Performance can be improved in many ways. Learning to write the best searches and using the narrowest time frames will improve search performance without any hardware modifications. Minimizing or refining search time field extractions can also increase search performance. Increasing the firepower of your present server might also be an option.
As far as benchmarks go, they can be really subjective things like:
Do searches seem really slow? Are your users complaining?
Or they can be more measurable things like:
Is it taking too long for data to get indexed?
Are certain searches slow and others fast?
Do you have a lot of concurrent users?
What is your daily indexing rate?
Are you planning on adding more users/data sources in the near future?
Adding a search head will not give too much of a performance boost since you are just moving SplunkWeb to a different machine. The way Splunk works splunkd does almost all of the heavy lifting. It indexes your data and it runs your searches, SplunkWeb just runs the user interface. Splitting up your indexing and searching across 2 indexers will give you the best performance increase since you are doubling both the indexing and searching power that way.
Check out one of our founder's blog entry on this topic: http://blogs.splunk.com/2009/10/27/add-a-server-or-two/
