The query/search optimizing game of Splunk is, compared to let's say an Oracle Database for instance, pretty weak. If you want to understand how and why different searches take so much time (or not), I strongly recommend to learn the basics of relational algebra (to have a starting point check out Wikipedia: https://en.wikipedia.org/wiki/Relational_algebra ). To give you a rough oversimplification of the basic idea: You want to keep your interim results you are operating on as small as possible as early as possible. Let's say you have some data with two fields:
Fields: "a" and "b".
And (to keep it simple here) let's say we know the following:
Field "a":
- 60% of all events got a="yes"
- 40% got a="no"
Field "b":
- 90% of all events got "b"="foo",
- 5% got "b"="bar"
- The rest is a wild mix of values or unknown.
And know let's say you want to search for something like this:
search a=yes | search b=bar (I use the pipe intentionally here)
If this search is performed by Splunk the following will happen (Hind: I'm not sure if Splunk wouldn't even optimize such kind of searches, how ever, I'm trying to explain the idea behind the optimation):
Let's say we search 10^6 (= 1,000,000) Events: search a=yes | search b=bar
- First: "a=yes": 1,000,000 events are searched, 600,000 events are left
- Second: "b=bar": 600,000 events are searched
- In total: 1,600,000 search operations
If we, how ever, switch the search order, we would have this: search b=bar | search a=yes
- First: "b=bar": 1,000,000 events are searched, 50,000 events are left
- Second: "b=bar": 50,000 events are searched
- In total: 1,050,000 search operations
Compared, Splunk had to search ~45% less events (assuming the search time is linear to the events searched 45% faster, too). How ever, please don't fixate on this simple example, just realize the idea behind it. As Runals already said, a good idea is to start by adding a index (if you have more than one) to reduce the interim results. Having some statistical information about your data (like in our example) may help, too. The idea, why the TERM(...) syntax could help, is because it's reducing the searches to a single search (and not breaking them up and joining the results afterwards).
Another idea to optimize your search is to use always projection before selection (those are terms from the relational algebra, seriously, learning that will help you a lot actually).
- A selection is filtering rows
- A projection is filtering columns
When using search query: Splunk will have to check all fields (columns) of your events. Whereas using a term like search text=query: would be
- 1st a projection (search in column text , ignore all other columns)
- 2nd a selection (select only rows matching query: )
If you check out the "Quick tips for optimization" (here: http://docs.splunk.com/Documentation/Splunk/6.2.0/Search/Quicktipsforoptimization ) you'll see basically the same (without an explanation why, which is much more important to understand).
... View more