Indexing only uses about 4 procs per pipeline (essentially 1 proc per queue). Therefore, looking at global CPU may not be a good measure. It is certainly possible to be CPU bound in the indexing pipeline, but is also common to be I/O bound. What filesystem is in use? I have seen some "unfavorable" results with some versions of XFS, which were improved by upgrading OS or moving to EXT4 (preferred). But as others have suggested, insure THP is disabled, and make sure ulimits are sufficient. ... View more

kaufmanm, Yes, you should be able to create a multisite cluster with on-prem and AWS (we have many customers doing this). What you can't/shouldn't do is create a single cluster with on-prem and AWS indexers in the same cluster (which I think is what you are alluding to). AWS should be a separate index cluster that can participate in the multisite cluster. But you would need enough indexers in AWS to handle the entire DC1 load... DC1 = on-prem AZ1 = AWS Then you can start replicating DC1 to the AWS AZ1, once fully replicated, you could begin sending all new forwarded data to the AWS AZ. This would allow you to shutdown DC1 entirely at that point. The bandwidth required during this transition would be the necessary bandwidth for your forwarded data (500GB/day) plus the replication of the buckets until you shutdown DC1 (initially very high due to replicating existing buckets, but then would drop down to just new buckets, roughly 250GB/day). You would also include any search results in your bandwidth needs - which will vary greatly depending on your environment. Bandwidth figures can be figured out by averaging this across 24 hours, but that won't take into consideration peak loads when there are spikes in forwarded data. But academically speaking you could get by with ~36Mb/s in bandwidth (assumes 750GB/day total transfer rate). ... View more

The short answer is "no". Splunk does not have the ability today to balance buckets by available storage on an Indexer. You can check out the docs for this here: http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Rebalancethecluster In the future, we are hoping to make cluster balancing more intelligent and do what you suggest, but I'm not sure when that feature will surface in the product. Instead, you will have to architect to more evenly add storage to your Indexers. ... View more

It's not that it is not "supported", rather, it's not a good idea for performance reasons. If you are doing it at a time that has low activity (particularly incoming writes), than it shouldn't be a problem, as long as you do what you need to and then remove the snapshot. As the author of that Tech Brief, I can tell you that all of the stuff in there are recommendations/best practices for scale and performance. Because VMware's snapshots create a new file on disk which Write I/O is routed to, the largest performance issue is the consolidation of the snapshot (where it has to merge all the changes from the snapshot). Also, by nature of backups, you are also putting a much higher Read I/O load on the disk in addition to Splunk's own Read I/O load. This is why I suggest you do it at point where activity is low (assuming there is one). The biggest benefit to snapshots is the fact that you capture hot buckets in a consistent manner (at least as it relates to a single Indexer). If you are on a SAN that offers snapshots that are integrated with VMware through the appropriate VAAI primitives, then most of the I/O heavy lifting is offloaded to the storage, and consistency is likely across all Indexers stored on that array. Of course, the easy answer is to use the Splunk recommended best practices for backing up 🙂 ... View more

Correct: https://wiki.splunk.com/Community:HowIndexingWorks ... View more

I'm not sure there is a reference in the docs to the 4 processor usage for indexing. However, there are four distinct queues and this process for the indexing pipeline. Each of those tends to leverage the better part of a CPU. That is the reason we say that 4 CPUs are used per pipeline. ... View more

I should add, if you go with the larger system, your expansion options are to add another identical system. While with 3 smaller servers, you could add a single identical smaller system. ... View more

Unless, you are dedicating resources on the virtual machine, it is unlikely that will perform as well as 3 smaller physical machines. In fact, you will incur a 10% performance penalty for indexing by simply running virtually (worse if there is contention for CPU, RAM, or disk). Also, a single indexing pipeline would use 4 CPUs per machine, thus 3 servers would have 12 CPU worth of indexing horsepower. As of 6.3 you can have up to 2 (max recommended) indexing pipelines which would give you 8 CPU worth of indexing horsepower on the "big server", which would be less than the 3 smaller servers, but would leave more CPUs available for handling search. If that meets your needs, it's probably fine. Personally, if I was setting this up, I would want the 3 servers with known dedicated resources, and the inherent redundancy that is associated with it. But if you are going for ease of use and simplified maintenance/administration, 1 server does fit the bill. Info on indexing parallelization: http://docs.splunk.com/Documentation/Splunk/6.3.0/Capacity/Parallelization ... View more

Default credentials for the forwarder is admin/changeme. Which you should change during configuration. ... View more

That's where the 2nd half of my answer comes in to play. bonnie++ Random Seeks per Sec are not equivalent to IOPs (nor do they claim them to be representative of IOPs). This is an issue I am working to resolve with our own documentation. Although the references in our docs don't use bonnie++ from what I am able to find. They actually discuss number of spindles, rotation speeds, raid level and expect IOPS based on that (which would be relatively accurate). This is an interesting serverfault post that summarizes things well. http://serverfault.com/questions/578250/interpreting-iops-these-bonnie-and-iostat-results Using iostat while bonnie++ is running would give you an IOPs reading. I would guess that will be more inline with what you are experiencing on the array. But it is not what Splunk means when it says "IOPS". Again, I am working to correct all of this. ... View more

This is usually like comparing apples and oranges. A disk array produces many more I/O operations for the same instruction than a single disk would because it is using RAID levels across many spindles. This does get aggreagted back to the front-end controller in most arrays. So a single read operation might hit 100 spindles thus creating a minimum of 100 IOPs. Also bonnie++ is looking at file I/O. This means it could be doing 8K reads/writes, but based on file system, block sizes, sector alignment, etc, this could get broken down to be multiple actual I/Os at the disk level. Most importantly, what bonnie++ calls Random Seeks per Second isn't a direct correlation to an IOP, which causes confusion because we (Splunk) say IOPs instead of Random Seeks per Second. Therefore, what makes for a successful storage platform for running hot/warm on Splunk is one that can consistently produce 800+ Random Seeks per Second using bonnie++ test with at file size of at least 2x RAM (to defeat OS caching). We have seen examples of SAN arrays that don't perform well with this test, but do perform well for Splunk, which has us working to provide a better testing harness for you to use in the near future. Stay tuned. ... View more

What is the command that you ran? That is a slightly different output than I am used to seeing. Also what version of bonnie++ are you running? ... View more

Avisram, the Tech Brief recommended above is definitely for a production environment. It is probably ideal to have a physical host with at least 20 cores and 256GB RAM for this environment. It's impossible to make a recommendation for storage based on the information provided. Would need to factor how long you plan to retain data in this environment, and work backwards from that (this site might help you: http://splunk-sizing.appspot.com/ ). If you plan to do multiple physical hosts for this environment, you can probably do smaller core count boxes. If you are building this to perform well, I would need further information to size this. ... View more

The Answers post you mention should work in a non-clustered environment. KVstore has it's own replication abilities, so setting the replication_host on the nodes in question should build a KVstore cluster. You will have to be careful not to have different searchheads that are not "coordinated" by a cluster, writing duplicate data to the KVstore. Also, it is possible to have a separate SHC for ES, although clearly that still leaves you with the same issue of uncoordinated KVstores between the ES SHC and the non ES searchheads. ... View more

For the sake of answering this for the future viewers. stdev(X) is using the sample based approach. Sample based is like taking a national poll, you would not poll every citizen to get an answer because it is too hard. Therefore, the larger result set you have, and the more varied the location, the more likely you will have an accurate mean. If you could poll everyone, you would have an absolutely correct answer, vs. an estimated answer. In most cases within IT use cases, we want to use the total population because we have all the data. To use the total population, you would use stdevp(X) - the p meaning population based. ... View more

Just to correct math write performance would be 420 IOPs. These numbers are ballpark but a good illustration. Thanks, Dart! ... View more

To answer the two questions, you originally asked, I believe the answer is simply, no. I have never seen an architecture that allows for a 2nd tier of indexers. The only thing you could potentially do here is to migrate data to frozen, send the frozen raw data to another set of indexers and reindex the frozen data there. And then you could search across both sets of indexers from your search head. This would be painful to manage, and I wouldn't recommend it. Alternatively, you can also add additional indexers to your current indexing tier/pool which will spread new data out over the entire indexing tier. But as you previously noted, if you use slower indexers, the performance of your entire indexing tier will be impacted. But again, the most cost effective and simple solution would be to add more storage to your existing indexers. ... View more

Check out the example in this doc: http://docs.splunk.com/Documentation/Splunk/4.2.5/Knowledge/Addfieldsfromexternaldatasources#Set_up_a_fields_lookup_based_on_an_external_command_or_script ... View more

Given your need for fast/reliable access to historical data, I would say, the best solution would be to add more disk to your current indexers. ... View more

good luck! Don't forget to vote 😉 ... View more

I should also add, that you should set sizes to leave some headroom on the filesystem. I.e. if you have exactly 100GB for the filesystem, you might set the max size of the index to 90GB. Modern versions of Splunk will ensure you keep at least 2GB free, but may shut down indexing if this limit is hit. ... View more

Not sure why Simon set that value originally, and I don't have the complete story here, so if there was a specific reason, you may want to find out from him. That being said, you are absolutely correct, homePath.maxDataSizeMB = 100000 will be triggered before you reached anywhere near 300 warm buckets. A good way to look at the way Splunk archives data is based on a series of triggers (size, age, count). If anyone of these triggers is tripped, action occurs. ... View more