So you want to trigger the alert action of your saved search if the total number of events exceeds some limit (like 100 events), but you want the content of the alert to contain the more standard "top"-like output. I did some playing around, and came up with two different options for you. Option 1: Use streamstats to get the total count, and use the "by" clause of top (just as a way of preserving the total count field). Name your "count" field with a leading "_" so that it doesn't show up on on the final output. Try something like this: index="errorlogs" | rex "EXCEPTION:\s(?<exceptiontext>.*)" | eventstats count AS _total_count | top exceptiontext by _total_count Then set your alerting condition to a search expression: where _total_count>100 Option 2: Keep your search pretty much as-is, and add some logic to your custom alerting condition: Search: index="errorlogs" | rex "EXCEPTION:\s(?<exceptiontext>.*)" | top exceptiontext For your alerting condition, use: stats sum(count) as total_count | where total_count>100 This approach is slightly easier since it doesn't mess with your main search, but it's also slightly less accurate. For example, the total_count will only include your top 10 counts; Since your adding up largest 10 values, this probably will not be an issue. It could work for your or against you, but I figured I should point out the slight difference in logic between these two approaches. ... View more

There are two different timestamps to keep in mind when looking at this kind of statistic: (1) the event's timestamp which is the date/time information that Splunk extracts from an event. Or (2), the time when the event was indexed. Unfortunately, there are some gotchas (or limitations) with trying to capture stats with either of these fields. Normally this isn't a big problem, but it's important to be aware of the differences and know the sometimes hidden assumptions behind either approach. If you create stats based on event timestamp, then you have no option other than to search over all of your events for an entire period of time. This can be resource intensive, and there are some tricks that help, like jrodman commented. One potential issue is if you have events that are sometimes delayed beyond your summary interval. (If your summarizing daily, and some events are delayed for a couple of days because down forwarder, then they obviously will not be included in your count because they didn't exist when the summary indexing saved search ran.) If you go with stats based on index time, than trying to directly get your stats using a search becomes very painful. This is because _indextime is not optimized for searching like "_time" is. However, you do have many more options with this approach, like using metrics as you pointed out. (Metrics can be quite complicated and are prone to misinterpretation. And I agree that trying a "maxseries" of 500 probably isn't going to look pretty. So this probably isn't an option for you.) Another approach that looks at index-time would be to capture buckets stats each day (probably in a lookup file), and then do a daily comparison and summary index the delta as the number of events that were indexed by host. At a basic level you could start with | metadata type=hosts , but you may need to capture this at a per-bucket level so account for buckets aging out. This could be complicated to implement, but it should be really fast, and you could conceivably run this at any interval that you wanted (weeks, days, hours, minutes). It's certainly nice not to have to search over all of your events. Update: I did some additional thinking about the metadata/bucket based approach. I thought perhaps | dbinspect would help, but it simply gives a count per axis (host/source/sourcetype) so there isn't a breakdown of the events per host. Having said that, this information is available in the Hosts.data files in each bucket, so it would be possible to write a custom search command to capture this info for this kind of stats collection. This would effectively handle the issue of bucket rotation, but wouldn't help with deleted events; then again there really isn't a good way to account for that in any of these situations. If the straight forward "sistats" approach is working; stick with that. But if your volume increases or you need a more creative solution, baselining and tracking changes to Hosts.data may be a more efficient solution to consider. ... View more

Sounds like a simple stats command is all you need. Try this: "Change Password Attempt" OR "attempt was made to change" | rex "Caller User Name:\s(?<Caller_User_Name>\S+)" | rex "Target Account Name:\s+(?<Target_Account_Name>\S+)" | stats count by Target_Account_Name, Caller_User_Name, host | sort host BTW, your search would be faster if you include some kind of source/sourcetype as part of your search criteria. Probably something like: sourcetype=WinEventLog:Security ... View more

You may find some information from this question helpful: http://answers.splunk.com/questions/3078/copy-an-index-on-the-same-splunk-instance Assuming your not on Windows, you can copy your data while splunkd is running, but your results may not be fully consistent. So some kind of files system or block-level snapshotting is ideal here to get a more consistent result. (For example, using a LVM snapshot). In this case you shouldn't need to bring splunkd down at all, or roll your buckets from hot to warm. Of course, it all depends on what kind of event loss tolerance you can handle. And if you're looking to do a one-time copy of a bucket, or use something like rsync on a ongoing basis (see the above link.) Stopping and restarting splunkd is certainly going to interrupt any running searches and put a temporary delay on any indexing. And if your actually bringing down splunkd then even your "hot" buckets will be consistent while the splunkd is not running. Also keep in mind that forcing a bucket roll, will NOT guarantee that all of your buckets are WARM, because splunk will immediately create new hot buckets for any events that are received between the time your script forceably rolls your buckets and the time splunkd is shut down. Again, the more details you can provide the more helpful the people here can be. ... View more