Presuming you already know what indexing buckets are... A splunk hot bucket is changed into an invalid_hot bucket when Splunk detects that the metadata files (Sources.data/Hosts.data/SourceTypes.data) are corrupt/incorrect. There are two types of incorrect data detected: the time ranges may be incorrect, or the event counts may be incorrect. We believe that the time ranges are usually at fault. An invalid hot bucket is mostly ignored by the index from this point on. Since we don't trust it, we don't want to put more data in it, and we do not search it. They are not currently (4.1.x) automatically recovered or automatically managed in any way. Invalid hots do not count as hot or warm for the index management considerations (max number of allowed hot, max number of allowed warm buckets). Thus, they will not negatively affect the flow of data through the system, but at the time of this writing (4.1.3) they can incur additional disk storage over what is expected, because the normal data will be stored in additional to the invalid hot data. In some cases this is inconsequential. There have been versions historically where splunk decided a bucket was invalid too early when it was still an empty directory. While a nuisance, there is no harm with that scenario. You can safely delete such an empty invalid hot (these were generated by versions around 4.0.3. If you are running such, please upgrade.) In other cases, real data had already arrived in the hot bucket before it was determined to be problematic. The corrective action for an invalid hot is to: Attempt to rebuild the metadata from the rawdata information (recover-metadata) If successful, rename the bucket as a warm bucket to rejoin the splunk index proper. The recover-metadata command is destructive. It will clobber the existing .data files in a bucket. I recommend making a duplicate of these files before running recover-metadata even if their only use may be for forensics purposes. To run recover-metadata, run `splunk cmd recover-metadata path/to/your/invalid_hot_5'. Hopefully, it tells you that it worked. If recover-metadata is successful, rename the bucket as it would normally be named (link to script forthcoming) and all should be well. ... View more

In short, use another mechanism to update them. The deployment server had planned functionality to provide this sort of update, but it was not fully fleshed out. An attempt to make use of this would be essentially an experiment. I do not think this would be reasonable in any production environment. If using the deployment server for this task is important to you, do push for it through the support channel. It may be that there is a small amount of additional work in polish, and then of course documentation, or it may be that much work remains. ... View more

Solaris has an nofiles ulimit of 256 by default. Other unixes have larger defaults. There's a variety of possible responses we could have to this sort of thing: throttle very active users refuse to allow more than N concurrent accesses scale either of the above in proportion to the nofiles ulimit Does this cure seem better than the poison? It introduces new problems. ... View more

The only change with soft raid is that you may want to invest a little more in CPU than you might with hardware raid, especially if you are considering using raid 5 (which we obviously recommend against for the warm buckets). The read and write rates of Splunk are typically more bursty than sustained, so the CPU increase is not likely to be very large. Software raid is typically a higher total performance solution, because the bandwidth of commodity processors grows much faster than custom raid solutions. The typical stumbling block is the boot behavior on disk failure. That's general IT stuff though, not terribly Splunk-specific. ... View more