I have a dashboard that is composed of a bunch of inline searches, the reason i dont use saved searches and schedule them is because they are probably hundred of thems across my multiple dashboards. Is there an option to tell splunk to keep an inline search in the dashboard cached for say 5 minutes. instead of scheduling it to run every 5 minutes. the point of this is I will have several users accessing this results and it would be great if splunk would run the query only if it hasnt been run in the last 5 minutes or if it doesnt have a a pre runed query in the jobs list.
What I notice is that some of the charts in my dashboard say <1 min and others will always say <1s but they are all configured in the dashboard the same.
Sadly thats kind of the point of savedsearches. This gives Splunk a unique reference to match identical searches. Even if you changed it so a search result was kept for 5-10 minutes, Splunk cannot match two search strings.
The fact that you have hundreds is really what savedsearches are for, to help making maintaining them easier and to improve performance. Best option is to sit down for a few days and start stripping the important or particularly heavy searches into saved searches, you can use the jobs menu in the top right to find the searches that take the longest to run.
The time is down to how old the search results are, if you have a heavily loaded system this time can be a bit wonky. Also if you have a lot of searches on one dashboard bear in mind that a search will lock a core and Splunk can only have a maximum of 8 cores, this means that one dashboard on a heavily loaded system can render in slow motion with perhaps a minute between searches completing.
Splunk is currently architected to only use a maximum of 8 cores. If you have more available the best practice is to actually run separate instances of Splunk across different port ranges to utilize the additional processing power available. As I understand it the next version will hopefully not have this lmitation and obviously you won't want to do this with indexer instances as unless you have the IOPS available it will just drag the system down.
If the different panels on your dashboard are drawing from the same data sources, time range, etc, consider using postprocess. While it will take some work to redo your dashboards (with hundreds of searches), it will help with the time discrepancies on the panels.