Custom Index

[mavilla]

I have one specific dashboard that I monitor with 7 tiles on it, there are times when the dashboard searches auto cancel and it is slow to load as well, once it is reloaded the results of the searched come back for a period of time then auto cancel again. Currently there is one main index for everything that's coming into Splunk, would setting up a separate index just for the searched in this specific dashboard fix some of these errors? All the searches in this dashboard need to be real time.

[richgalloway · Accepted solution]

First, it's rarely a good idea to put all data into a single index.
Second, putting the data into separate indexes probably won't help as the limiting factor is number of search slots (CPUs) available at the moment.
Third, a dashboard does NOT need to be real-time. Real-time searches should be used when the response will be real-time. For people watching a dashboard, a 1-minute or 5-minute interval is sufficient. By using real-time searches, your dashboard is tying up 7 CPUs on the search head and all indexers.

[mavilla]

Thank you for your answer Rich - just so I understand this correctly, you're saying that since I have 7 tiles on this dashboard and each of them is searching in real time, each tile is taking up one CPU to perform the searches? For context, our Splunk server is running 16 CPU cores (virtual), I see your point about the entire dashboard being in real time.

[richgalloway]

Yes. Seven real-time tiles/searches take up 7 CPUs, not only on the search head where the dashboard runs, but also on all indexers.

[mavilla]

Our search head and indexer are hosted on the same server, we've already identified this as being an issue but didn't know it was this much of an issue. Thanks again