Splunk does not neatly fit into these two words. When data is arriving in a streaming fashion to an indexer, it processes that data and then writes the processed data to disk in a streaming fashion, but also streams the processed form of that data to N other indexers. All of these indexers commit the data to disk continuously. The streaming target indexers respond with information about what items they've been sent have been fully committed, and when the initial indexer has fully committed and gotten notification of full commits from its peers, it can tell the data provider that the data has been committed. This works because we don't have a single integrated structure (conceptually) like an RDBMS. There aren't updates to our database. We just build small (10gb) quanta of duplicated containers in parallel. If there is error recovery, the total IOs may be higher, but they can still proceed continuously without slowing any part of indexing past I/O limitations. This design, not being based around a transaction->commit model means that you can end up with data duplication in some error conditions, (data provider is talking to indexer that crashes, so sends data elsewhere). However that is a fundamental problem, because the data provider is not a database application making use of a transaction api, but rather an application that wrote to some log files it will rotate out of existence without asking us, or an application which sent some bytes over a socket and is done with them. There are partial solutions to that but they are performance-prohibitive at the data volumes involved, and nothing is going to make the data over a syslog-tcp socket or in the tcp buffers safe on crash. So it's a different sort of solution for a different kind of data flow. But the short version is that replication does not require any sort of stalling for synchronizing, but oversaturated disk or network links are concerns. ... View more

Question 1 - Warm vs Cold is just to allow for split storage, nothing more. Typically fast vs cheap but it could be ad-hoc for "some storage" + "some more storage" as people grow. Question 2 - maxDataSize is the ceiling size for a bucket. That's it. It thus influences transitioning from hot state to warm state, or "when we stop putting data into the bucket", which is really what that means. If you're controlling warm by count, that indirectly influences when things leave warm for cold, but I think most users these days would rather control warm by size (homePath.maxDataSizeMB) or put the whole thing in a volume and have that control by size. There are no controls to roll from warm to cold (swap filesystems) based on time. I can see why you might want to, but usually volumes achieves the same goal, since it will roll out the oldest bucket of everything it contains based on age. Question 3 - If an index contains a non-hot bucket where all data in the bucket is older than frozenTimePeriodInSecs then it is frozen (by default, deleted. modify with coldToFrozenDir / Script). Thus a low value like 24 hours (24 * 60 * 60) will typically cause a bucket to live for 24 hours after rolling to warm, assuming the data arriving into the bucket was data for now. If the bucket was receiving only old data, it might vanish immediately on rolling. If the bucket is receiving future data, it could live for quite a while, depending upon how future. Obviously future data is usually a misconfiguration or misbehavior. ... View more

Essentially, it means the 'modular input' that gathers Windows Event Log data sent in an event that did not say what index it was for. Since all inputs (scripted, modular, file monitors, etc) are supposed to tell the splunk data processing pipeline what index their data should land in, this is considered an error. Most likely (no promises) this is an error in Splunk-provided code, and not a configuration or user error. It seems unlikely that it relates to the service ceasing to run, though it is possible. Splunkd has handled data with no index key many times in the past without crashing. Essentially this is really a technical support problem, though if the misbehavior was a one-off it may be difficult to gain traction / not necessarily worth your time to work through. ... View more

In general, Splunk does not recommend sending data from newer forwarders to older indexers because you can run into hangups like this. There was an exception clause for Universal Forwarders that created during the rollout of the Universal Forwarder functionality, but the utility there has long since passed. Basically, when the instrospection functionality was built, we decided to place its data in its own index, because of concerns regarding storage, and the potential for introspection to overwhelm _internal, or because administrators were likely to have different opinions on the amount of space they would be wiling to allocate to one category or another. Thus we added the introspection index to all versions of splunk after this feature. However, when forwarding data to older indexers, they do not have this index, and thus the data cannot be stored in the index. For about 5 years, we have produced highly visible messaging when indexers are receiving data for an index they do not have, because the indexer has no way to handle the data properly, and it usually indicates a big misconfiguration problem. In this case, you can simply create an index called _introspection on your indexers, or upgrade them to 6.x. If you do choose to create your own index, you may wish to revisit your index when you do upgrade to 6.x+ to rationalize your settings vs the defaults. I suppose an alternative would be to disable introspection data acquisition on all the forwarders, but introspection data is good troubleshooting information, and this sounds like a larger administrative burden than my first proposal. But it's your choice. ... View more