There's not any exposed controls to filter the so-called bulletin board messages. I think internally some messages are filtered, but it's not important because we obviously haven't filtered everything you would tend to want. There's an ongoing conversation internally around improving both the bulletin board and license messaging. I will point people to this answers link, but explicit bugs/ERs around this topic are a good input. ... View more

Rough first take: In the <indexname>/db directory, delete the file .bucketmanifest In the <indexname>/db directory, create the file (0 bytes works) meta.dirty If we get into goat sacrifice territory, try also deleting .metamanifest. Step 2 should render that unnecessary. These files and their associated data should get rebuilt on need by search activity. ... View more

Mostly, search-time fields have superior performance to parse-time (indexed) fields, regardless of whether they are explicitly configured. When running a search that includes a term such as fieldname=value , Splunk will treat this as a search-time field by default, unless fieldname is explicitly configured as an indexed field in fields.conf. This is true both for configured fields (delimiters, regular expressions) as well as for automatically identified fields where, eg you have fieldname:value in the text of your event. We call this automatic handling code auto-kv for automatic key-value extraction. The Splunk search machinery presumes that value will be present in the events as an indexed string, and will apply the same mechanics to filter the events as if you entered the string directly without the fieldname or equals sign. For most patterns, this offers all the performance advantage of a parse-time field, and none of the penalty. The tradeoffs are discussed in more detail in "About indexed field extraction" in the Getting Data In Manual. In all cases, the post-filtering is applied to the (hopefully) small set of events that actually contain the value string, by applying any extraction mechanisms, then testing to see if the field has been created containing the desired value. Ideally the index-based filtering is the most important factor in the speed of your search, but there are cases when search-time extraction must be applied to a large percentage of events. For example if almost all of your events have the word xml but only a small portion have this value in the storage_format field, the speed of extraction becomes important. Delim-based extractions are quite fast. Auto-kv are quite fast. Regex-based extractions are slower. Sourcetypes with a very large number of regexes or very inefficient regexes can be slower still. ... View more

This happens when an alert is created in manager, instead of in the search app. We neglect to create the viewstate ID (vsid) that the UI wants to figure out how to show the search. As a result, the UI fails saying "hey, I don't know how to show you this search" in its own inscrutible way. Workaround, create a vsid for the search (someone will have to provide some more detail here.. I don't know the steps). Solution, use Splunk 4.0.10 or later which doesn't muck this up. ... View more

On a reasonable OS and filesystem, I think you can get pretty reasoanble behavior for a file as well with small (under 4k) writes where you set the file to append mode, unbuffered and flush after each write. And your apps should really be doing that with logfiles anyway, if you want to find out when they crash what happened. So I'm not sure the big deal with FIFOs is atomicity, though if you're sure of the behavior, go for it.. that sort of thing is pretty well outside the Splunk boundary. Where I've foud FIFOs useful is when writing auomated inputs, like scripts. The FIFO acts as a flow control for your program, which lets you have a pretty good idea of when and how fast that data is getting into Splunk. It also allows you to be pretty lazy in your script authoring without much of a problem. The most obvious downside is crashes. If the system crashes, or if splunk crashes, some data will be lost. Splunk can't put the data on disk from the FIFO before it reads it, and the OS isn't going to provide a backing disk store. Even if the source app has the data, there's no generic protocol for the program and splunk to renegotiate the position. The second problem is debuggability. If something's going fishy with your datastream, the FIFO offers no clues. It's hidden from view. ... View more

I'm not aware of any reason to use multiple receiving ports for splunk-forwarded data. It's possible that at very large numbers of forwarders (say several thousand and up) there may be scalability issues that are mitigated with multiple receiving ports. Historically we have identified some problems of this shape, but the known issues have been addressed by design. When sending non-splunk data, such as syslog, into a UDP or TCP port, there are convenient reasons to use multiple inputs, because Splunk can apply default to the data at the input level, such as host, sourcetype, and so on. However, this should not be necessary and is not really possible in the forwarded case; the forwarder has already labelled the data. If you wanted to have both SSL and non-SSL forwarders connecting, that would require two ports. ... View more

We store the timestamp values in UTC always, and just display them in the localtime where the server is configured, so changing the timezone of splunk should not damage the accuracy of the timestamps in any way. What might be trouble is if the incoming log datastream does not declare its timezone, and thus the newly arriving data is interpreted differently from the historical data (one of them is probably wrong!) ... View more

It's certainly true that if splunk encounters 'someapp.log' without configuration, likely to create a new sourcetype called 'someapp'. Later, as the file rolls, splunk may not be able to correctly guess that the new rolled files are the same, and create a new sourcetype 'someapp-1', and then 2 and so on, as you say. However, IIS gets these sourcetypes for another reason. IIS is a sourcetype with positional field names in a header at the top of the file. However, since each file lists the fields present, Splunk assumes that not all files of this type will necessarily have the same list of fields. Therefore a new sourcetype is generated whenever the list of fields must be stored, and the list is inserted into a field extraction configuration for each sourcetype in turn. This works fine in a simple splunk environment, although it does look a bit confusing. However, because it creates configuration at index time intended to be used at search time it can break in distributed search environments, or in situations where data is forwarded after it is parsed. Incidentally we have a proposal to make searches work for all 'sourcetype=iis' which is to add the configuration 'rename=iis' to each of the autogenerated sourcetypes. This can be done manually for now, but I hope this starts happening automatically in a release in the near future. ... View more

4.0 doesn't have terribly good log events for alerting. You can see that the search was run, but not that it was run by the scheduler, so you cannot differentiate between manually-initiated and schedule-initiated searches. You can see the python event if the search eventually fires the email sending command sendemail.py, but that only will catch searches whose conditions were met, and which were configured to send email. In 4.1, all scheduled searches are explicitly logged, as well as the result (conditions met / not met). If a search would have run but was not for some reason, this is also logged. There are some built-in status views that try to give useful reporting on this data, but you can build your own slicings of it. ... View more