The .sentinel files are used on Splunk for win32/win64 as part of the locking strategy, to ensure exclusive access to buckets for certain types of modifications. Windows doesn't allow locking calls on directories with the same semantics as we desired. They are mostly useless on a unix-based system, but in the interests of the index data format being able to traverse platforms are produced nonetheless. Best practice is to move the .sentinel with the bucket it goes with. ... View more

Nope. It's an internal discussion topic. I'm in the sustaining group, charged with removing defects and making things more supportable. This is one of the things we would like. It's a bit tricky in that SQL datasets are a bit more expectable than splunk datasets, so something like a plan or explain would take more interpretation for Splunk than for SQL, but still it would be a good tool for aiding in performance analysis. Things you can do right now: run a search and review the search.log in var/run/dispatch/ it will have some information about expansions, so that you can see how tags, eventtypes, and so on are behaving. review http://www.splunk.com/wiki/Community:PerformanceTroubleshooting , which down in the 'anatomy of a search' will give you a rough idea of how the machinery works, so will give you a good idea how to proceed from an expert perspective. work with support on specific searches you want to go faster If you have specific components of the information that a plan or explain would provide that are most crucial, we'd love to hear about them, and the shape of the problems you face that make these important. Any assistance our customers can provide in scheduling decisions is very much appreciated. I can copy things into formal enhancement requests, but ideally those kinds of things arrive via the support channel. ... View more

There's no fixed requirement for when lookup should be invoked. There are two possibilities here. One is that your lookup is keyed to some fields that aren't available post-stats. Another is that the lookup operator presumes some fields which aren't available post-stats. Perhaps you should provide a bit more about how the lookup is defined? ... View more

I suspect the the actual pdf generation step is missing here. sendemail's only responsibility is to mail out the pdf. I do not think the action to fire the pdf generator is exposed to the search language at this time. ... View more

Sparc is a low-performance choice for Splunk indexing and search. The reason is that Splunk is CPU intensive as well as I/O intensive (and depending upon search patterns can require a good deal of memory as well). Moreover, typical Splunk tasks for a single request are not very parallelizable, and will be typically bound by single-cpu-core performance. For Splunk workloads, a typical x86-64 modern server-class core will deliver as much as ten times the performance as a modern sparc server-class core. The Ultrasparc T1 and T2 "Niagara" CPUs, such as the one linked, are particularly ill-suited. These CPUs are designed for high efficiency and high parallelism, but without high performance per thread. This makes them good at loads that are individually light but highly parallel, such as high volume web sites. Essentially they optimized loads where task switching could be a large portion of the cost, or when there is a high penalty for small latencies. Splunk has only moderate parallelism, requires high performance per thread, and does not have the costs or problems this arch optimizes for. Although Splunk has a web interface, it is not serving static documents like a web server, but performing significant computation and I/O to run searches. To a lesser degree these same issues will be a concern on Itanium or POWER systems, but there it's more a matter of the price per performance ratios being quite different as opposed to the raw performance. In general, x86-64 or amd64 CPUs are the recommended architecture / CPU types for Splunk. As a general guide, take a look at http://docs.splunk.com/Documentation/Splunk/5.0/Installation/CapacityplanningforalargerSplunkdeployment Note that Splunk scales well horizontally, so a number of cheaper machines in parallel are often desirable over one larger system with many cores. Don't forget to take all this advice in context. If you're indexing 100MB a day and have 4 users, performance considerations become much less essential. ... View more

Feel free to brave, if you want. Armed with the idea that cli.py->main() is the entry point from the splunk executable into the python code, most of the rest of it is fairly straightforward python, but a lot of what it's trying to do is quite specific to splunk. If you've got time to kill and want to debug a bug you're seeing, you can grab the logger we use and coerce it into printf style debugging. Sounds like a waste of time to me, though. In this case I presume that files_to_export is set in a conditional path that was traversed in our testing, but not for you. Okay, actual fail here is an incorrect exception catch combined with an incorrect exception handling. Either the glob or the filter tossed an exception (that the migration code hid), and then it didn't handle the lack of a variable correctly. You could edit migration.py: def suggestExport(bundle, metadata, subdir, type, is_correct_file_type): base = bundle.location() try: candidate_files = glob.glob(os.path.join(base, subdir, '*')) should be hacked to def suggestExport(bundle, metadata, subdir, type, is_correct_file_type): base = bundle.location() search_dir = os.path.join(base,subdir) if not os.path.isdir(search_dir): return False # no objects -> no export needed try: candidate_files = glob.glob(os.path.join(search_dir, '*')) I'm filing the bug. ... View more