There is no single good answer to this question. Generally, indexed fields cause additional overhead in terms of storage size, can - if bloated - counterintuitively have negative impact on performan...
See more...
There is no single good answer to this question. Generally, indexed fields cause additional overhead in terms of storage size, can - if bloated - counterintuitively have negative impact on performance and for straight event searches do not give that much of a performance gain versus well written raw events search. Having said that, there are some scenarios when adding some indexed fields helps. One is when you do a lot of summarizing on some fields. Not searching but summarizing. Then indeed tstats is lightning fast compared to search | stats. (OTOH you can usually get similar results by report acceleration or summary indexing so indexed fields might not be needed). Another case is when you have a lot of values which can appear often in multiple fields. Splunk searches by finding values first and then parsing the event containing those values to find out if it parses out to given field. So if you have 10k events of which only 10 contain a string "whatever" and out of those ten nine are values of a field named comment, a search for "comment=whatever" will only need to check 10 events out of those 10k and of those 90% of considered events will match. So the search will be quite effective. But if your data contained the word "whatever" in 3k events of which only 9 were in the comment field, Splunk would have to fetch all 3k events, parse them and see if the comment field indeed contained that word. Since only 9 of those 3k events contain that word in that right spot, this search would be very ineffective. So there is no one size fits all. But the general rule is that adding indexed fields can sometimes help and it's not a thing that should never be used at all but should be only done when indeed needed. Not just added blindly for all possible fields in all your data because then you're effectively transforming Splunk into something it is not - a document database with schema on index. And for that you don't need Splunk. And if your SH is already overloaded, that usually (again - as always, it of course depends on particular case; yours might be completely different but I'm speaking from general experience) means that either you simply have too many concurrently running searches. And creating indexed fields won't help here much. Or you have badly written searches. (which is nothing to be ashamed of; Splunk is easy to start working with but can be tricky to master; writing effective searches requires quite a significant bit of knowledge).