I think the simple answer is that we don't yet have real-time search based alerting, so probably most of your existing searches will want to stay as-is. However, there are some searches which might be useful more as an investigative realtime search than a periodically generated report, etc. But that would be highly specific to the searches and the user stories. ... View more

Here's a rough document I wrote around this topic, but it's more about the configuration data and being sure it will work in the new environment, rather than the live handoff: http://www.splunk.com/wiki/Deploy:Migrating_a_Splunk_Install When you rsync you might want to omit traversing the hot buckets, as you could end up doing a large amount of I/O for temporary files, ie. transferring the same data many times. I recommend the hulahoop approach of parallel installs. It gives you much more flexibility to deal with problems and ensure correctness. Once you have your parallel install proven to work nicely, you can run clean eventdata on it, and begin bringing data over from your old system. You can merge the indexes, so long as you are sure to cut it off around the time that you brought up the new indexer, and avoid id collisions. I wrote a script in an attempt to ease the merging of indexes (handle the id collisions), but it isn't publically facing yet. It's not a hard job, just renaming the directories so the third number doesn't collide. ... View more

Windows does not have a gid or uid, thus these fields come in as "unset". The equivalent concepts on Windows are owner and ACL list, which cannot be expressed as two numbers. If we don't currently capture Owner, that's kind of a shame. It shouldn't be too difficult to put that in a field. ACL list is pretty tricksy as it can be arbitrarily complex. Should at least be an ER to capture that stuff. ... View more

I believe there were some cases that were too complex to safely automatically handle, and the work was not scheduled. In common situations, the format is identical, as you point out. Outputs.conf is indeed a global configuration still, so the app space it lives in is not really essential. ... View more

I don't think there is a good way to achieve this at the present time. The best current response is to consider a pool of auto-lb systems to be one vesrion of the truth, so that if you have say four splunk instances, organized into two pools, then you can bring down one server from each pool at any given time and still have a complete duplicate set of data available in both pools when the nodes return. ... View more

You'd have to more or less roll this yourself, eg, run a search every 5 minutes and look at the last x recent minutes to see if the number has changed drastically. However, in the splunk world, you often tend to run your indexing system somewhere near capacity, so sometimes during a spike it can go over capacity, causing lag in the indexed data, which might make your volume appear to go down. Options: You could use 4.1 when it's released Real Soon Now(TM) run a real-time search on the forward's internal logs of its aggregate volume. This represents continuous load, but it might be that high. Would require evaluation. You could use a frequently run search on your actual dataset over recent time, say 15 to 5 minutes ago, and alert if it rises drastically or droops drastically. For this type of search goal, you don't really even need full coverage, a sampling might be sufficient. eg a search of sourcetype=foo host=bar | stats count as event_count | eval event_count>50000 or event_count<200 This would emit one event only if they count is outside the threshhold, so you could make your alert condition be more events than 0. ... View more

I think somehow answers has killed your formatting. I'm guessing this looks like: ---- Timestamp: 9/22/2009 4:50:04 PM Message:MPP Lab says the app... Category: blah blah ---- Timestamp: ... Seems like a cruel joke to have this big logging infrastructure that then creates timestamps in an ambiguous date format. But that aside, There's various ways to handle this. You can split the events by using the BREAK_ONLY_BEFORE and similar options, or you can use a clumsier but faster approach of changing our LINE_BREAKER from the default of ([\r\n]+) to something like ([\r\n]+----[\r\n]+) At that point you'd add in SHOULD_LINEMERGE=false , and splunk will then enforce that each block of text between that four dashed line will be treated as a single block of text. To extract the fields, you can use a repeat-match searchtime regex, in multiline mode, that matches something like ^([^:]*): (.*)$ with a FORMAT of $1::$2 Ie. for each line, the text before the first colon is the field name, and the text after the colon and the space is the value. This is all assuming the file is a text file, and not a complex binary format that happens to have text blobs in it somewhere. ... View more

Monitoring server.log only can work well, but there's an unavoidable race where we can miss the end of the file. For some users this doesn't tend to occur or they don't mind missing a few lines. I generally recommend monitoring server.log as well as server.log.1 This issue is generic to all rolling logfiles. Splunk 4.0 and earlier wait for the file to become 5 seconds stale before closing and re-opening it (which is how the roll will get handled). If your file rolls multiple times in that 5 second window, some files will be missed entirely. You can tune the timebeforeclose value in local/limits.conf, but there can be a performance penalty as our setup and teardown of file input streams isn't our best optimized behavior. If you have a relatively fixed number of file inputs, and changing the logging behavior is undesirable, it might be best to kick up max_fd in limits.conf to a value larger than your input count (say 250 for 200 files), and then set dedicatedFd on for your inputs pointing at those specific files. This means splunk will more or less always be trying to keep those files open. At this point you can drop the time_before_close to a value like 1, and hopefully this will catch every roll. Realistically you probably want your files to roll less often than this. Having your data expire from this world in 400 seconds means you'll likely lose data during spikes, or brief splunkd downtimes, such as upgrades. Maybe the total datarate is just so high you can't keep data longer than this? Note that 4.1 rewrites the file acquisition code, so that the worst-case time to acquire active files shrinks drastically, but 2 seconds may still be stretching it for a fairly busy forwarder with many data sources. ... View more