Edit your savedsearches.conf file and set the dispatch.ttl value. The default value is 2p which means 2 times longer than the scheduled interval of your search. savedsearches.conf : [my_very_long_and_intensive_savedsearch_name] .... dispatch.ttl = 10p .... From the savedsearch.conf docs: dispatch.ttl = <integer>[p] Time to live (in seconds) for the artifacts of the scheduled search, if no actions are triggered. If an action is triggered the ttl is changed to that actions's ttl, if multiple actions are triggered the maximum ttl is applied to the artifacts. For setting action's ttl refer to alert_actions.conf.spec If the integer is followed by the letter 'p' the ttl is interpreted as a multiple of the scheduled search's period. Defaults to 2p. ... View more

First, what's the time range of your summary indexing saved search? You mentioned that it's running every 30 minutes, but are you searching from -30m to now? If so, I think the simple fix is to do something like this: earliest=-35m@m latest=-5m@m This will give you a 5 minute window for your logs to get picked up by splunk and indexed. You may be able to get away with a shorter window, you can play around with that. Generally 5 minutes is enough to cover even a splunk restart and possibly a quick reboot. In terms of how many events, yes there currently a limit of 10,000 events per this post. Take a look at how many results you get after the sistats command? Some stats functions (like distinct_count ) can generate a massive amount of events if you have a field has a large number of distinct values. Sometimes the number of summarized events is greater than the number of input events. If you find this to be the case, then you are not gaining any performance improvements using summary indexing and you should review your indexing approach. ... View more

Hmmm, what timerange are you running your search for? When I run uname -a I get the following output: Linux linuxbox2 2.6.24-27-server #1 SMP Mon Feb 22 19:42:25 UTC 2010 i686 GNU/Linux Make sure you are preventing that kernel build time from being picked up as your event's timestamp. I believe you can do that with the following props.conf entry: [uname] DATETIME_CONFIG = CURRENT ... View more

This is because your events are not actually stored in all cases. (If you switch from the results view to the "Events" view when you open a job, you will see that the events are missing there too.) You can manually change this by adjusting the dispatch.buckets value. This is 0 by default for saved searches (and 300 for interactive searches). This is 0 for saved searches beacause you don't need interactive feedback as the job runs, which does allow it to run faster in the background, the downside it that you don't get any timeline info and the actual events are not stored. If you want to change this for a specific search, you can find your saved search in savedsearches.conf and add an entry like this: [your_saved_search_name] ... dispatch.buckets = 300 ... Alternately, I often just find it more convenient to re-run the search. All the parameters are already set for you, just just have hit the green search arrow. (Of course if this is a big search, than this can be an expensive operation.) From the savedsearches.conf doc: dispatch.buckets = <integer> The maximum number of timeline buckets. Defaults to 0. ... View more

You should be able to do something like this: host=squid* source=*access.log | stats count by domain, user Or another possible grouping would be like: host=squid* source=*access.log | stats count, values(domain) as domains by user ... View more

Is this log format that you control? In other words, are you asking about best practices for writing out log messages in a format that splunk will handle natively, or is this just an example of what you have to deal with that somebody else is writing out? I'm not sure you can escape the quote, but I know that sometimes splunk handles this better: 2010-01-01 00:00 key="value", key2="value2", key3="value3" If you have a comma between your events like this, then you may be able to use splunk's delimited field extractions. (I'm borrowing this from Splunk's built-in stash sourcetype which is used for summary indexing events which are automatically formatted to look like the key/value message shown above.) The key to this approach is the DELMIS = ",", "=" entry. Sample props.conf : [my_source_type] KV_MODE = none REPORT-my_fields = kv_comma_sep Sample transforms.conf : [kv_comma_sep] DELIMS = ",", "=" CAN_OPTIMIZE = false You may also find Splunk's Common Information Model wiki page helpful. To handle these at search time, you will probably need to use a custom field extraction. See Overview of search-time field extraction in the docs. ... View more

The summary indexing process will use _time for the event's timestamp if _time is a field that exists in your results. (As per How does summary indexing handle time?.) But in the normal case of using some stats-like command, you don't often keep the _time field around so the summary index process falls back to the time of your search. If you want to use one of the stats commands and you want a better time breakdown, you could look at using bucket command and set span to something less than the interval of your saved search: ... | bucket _time span=5m | stats avg(thruput) by _time host (You may also find sitimechart helpful here, but I've generally avoided all the si* helper commands and handled the funky statistical corner cases myself rather than let splunk do it. I've seen some of the si* command produce more "summary" events than I had input events... which is a step backwards!) With bucket or (si)?timechart , you will still not have the exact _time of the original event, but that's rather central to how summary indexing works. I suppose you could do a | stats min(_time) as _time by field but you will still only keep one timestamp from your groups of events... the bottom line is that you can't keep the exact same timestamp of all your events without duplicating all your events, which then defeats the purpose of summary indexing.... In terms of keeping sourcetype . You can't (or should) do it. In splunk 4.x, the summary indexing process does now set source to the name of your saved search. You still have a copy of the savedsearch in the event itself called search_name, but searching against source (since it's one of the primary indexed fields) is really fast. So I would just suggest that you leverage that instead. You still don't have a great drill down option with this, but it's possible. (You can let the sourcetype field go to your summary index, but it get's renamed orig_sourcetype which I suppose you could then leverage for drilldown purposes.) I suppose you could make a TRANSFORMS entry on the stash sourcetype that would look for orig_sourcetype in your event and then assign the sourcetype to that value, but that just seems like a bad idea.... ... View more

There are two approaches that you could take. These have a very different performance implication if you have lots of events, but if you don't then either one should be pretty fast. Transaction search: sourcetype=my_source_type ... | transaction fields="host,TransactionId" | search "my-xyz-error-message" This approach will first group all of your events into individual transaction events based on unique combinations of host and TranscationId. You may also want to look into the other transaction options as well, such as maxspan which is helpful to limit the overall time range of your transactions, which is especially important if TransactionId repeats. You can make this approach work better if you can filter out unwanted messages in your base search. For example, if you have many known-unhelpful messages (such as debug messages or startup messages that can be ignored) you can save resources (and process a larger number of transactions) if you can filtering out a large number of unwanted messages. Use a sub-search: sourcetype=my_source_type [ search sourcetype=my_source_type ... "my-xyz-error-message" | fields + host, TransactionId | fields - _* | format ] This approach will take the inner-search (This is a "subsearch" which is looking for any TransactionId and host combination that has your error message) and builds a dynamic search for the combinations that are known to have this error. (Note there is a 100 event limit on the on sub-searches which may prevent this from being a valid option for you.) With this search, events will be show individually. If you prefer the combined transaction look you can do that here too. Simply stick ` | transaction fields="host,TransactionId" at the end of this search. For other tips and tricks, I recommend the following: Browse around on this (answers.splunk.com) site and the docs. Check out the commands shown on the Search command cheat sheet ... View more