Continuous delivery is a method of developing and releasing software in a manner that minimises downtime and thus allows swiftly releasing software to production outside of normal downtime window, as well as testing it in exactly the configuration it would be in prod/live. You run two instances of the same application, with access to the live (light) instance being directed through a router (in our case, an apache redirect). New code is released to the non-live (dark) instance and testing is conducted there. Once testing has passed, the router is changed to direct traffic to the dark instance - promoting it to live. This can happen many times throughout the day at any time. A more detailed explanation of continuous delivery can be found here: http://martinfowler.com/bliki/BlueGreenDeployment.html We have Splunk dashboards monitoring our application logs providing views into things like numbers and types of errors, transaction durations etc. We are planning to move our apps from normal single instances to the design outlined above. However, this causes a problem for us in Splunk because we need to maintain charts for only the live (light) instance of the application automatically (the dark instance will be filled with errors as its mostly untested, and testing traffic will pollute the useful live info). There are three problems: The mechanism of determining which instance is live needs to be automatic because the instances can be switched to live at any time by our application teams (who do not have any Splunk access). Because the instances need to be identical, there is no way from the log event that we can tell which is acting in the live role. I can't restart Splunk many times in the day as it will effect other users, so any solution that requires restart Splunk server is not practical (restart forwarder on app server is OK). This seems like a deployment mechanism that many web based companies with high availability web applications might use. Does anyone have any ideas how we can achieve continuous monitoring of only our live instance from within our dashboards? ... View more

I often get asked by app teams "how can I see all the log files that are being monitored for my app servers" (they don't have access to see their forwarders inputs.conf and I'd rather not do it for them) or from IT security "how can I see all the sources of data that we are monitoring and where they are being monitored for the whole environment, so we can make sure we are covered". I have not been able to find out a good way to do this so far at all, other than to do a search like: * | stats count by host source ...which is ridiculously slow with our massive volume of data, and of course could miss out any infrequent sources that weren't actively logging during chosen search time range. As far as I can tell, I'm not looking for a "| metadata" search here as that can only give me a list of all sources (no idea which forwarders they actually come from) or a list of hosts (no idea what they are actually logging), not a combination of the two. ... View more

Hi, I have created a new app for one of our teams. This includes a new role dma, and new indexes dma_main and dma_summary. The dma role has been set up to search the main;summary;dma_main;dma_summary indexes by default. However when users of the app (who have the dma role) try to create a new summary scheduled search, they only get a single index listed in the "select the summary index" dropdown on the "add new" search page. They cannot see the dma_summary index that they should be using. Why is this? The dma role has been set up to inherit capabilities from "power" and "user" roles. Their inherited capabilities are listed as: change_own_password get_metadata get_typeahead list_inputs request_remote_tok rest_apps_view rest_properties_get rest_properties_set rtsearch schedule_search search There doesn't look like anything else in the capability list that is relevant. Please help! Cheers, Glenn ... View more

I'd like to user indexer acknowledgement in my HA setup when forwarding from a primary indexer which receives events from forwarders, to a secondary indexer (despite the horrible proliferation of duplicate events it can cause, but that's another issue). I'd like to know whether the queue or list of unacknowledged events maintained on the primary indexer will persist if the primary indexer is restarted (while the secondary is still unavailable). If it doesn't, we could easily lose the queue and have gaps in our secondary index, breaking HA. ... View more

Hi, I am doing a major overhaul of our Splunk infrastructure from a clone pair of standalone indexers to a multi-indexer, multi-dedicated-search-head (not pooled), deployment server. In an attempt to reduce the level of manual configuration of the new servers, I've reduced the process to simply install the rpm, run a couple of setup commands and point Splunk at our deployment server (which would be done by puppet or equivalent). The deployment server contains all the "system" config for the various different Splunk roles, including web.conf, outputs.conf, inputs.conf, indexes.conf, server.conf, authorize.conf, authentication.conf, limits.conf. The freshly installed Splunk pulls down all its config, restarts and is all configured correctly. Great! ...But I just came across this piece of documentation that advises caution when using deployment server, so I thought I'd get some second opinions. http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Updateconfigurations#App_management_issues Is what I'm doing a good idea (apart from the not insignificant risk of messing up serverclass.conf and destroying everything)? Is there a "best practice" document for managing this kind of setup while reducing manual intervention? Cheers, Glenn ... View more

I'd like to take advantage of search head pooling mainly because of its handling of scheduled searches (I'd be happy to go with rsync-ing data, but can't come up with a good way to deal with turning off all the schedules). Plus there are other benefits. However, for ease of installation, we set up the search heads' system config (such as authentication.conf, web.conf, etc.) via apps pulled from the deployment server. They do not live in the main $SPLUNK_HOME/etc/system area, but under $SPLUNK_HOME/etc/apps. We also have a set of normal knowledge apps with views etc. I enabled search head pooling in the normal way, copied the user directories and only my knowledge/view based apps to the shared storage. Because of the line "Important: You can choose to copy over just a subset of apps and user subdirectories" in the documentation http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Configuresearchheadpooling I expected that apps in both the shared storage and $SPLUNK_HOME/etc/apps would show up, but they don't. Only the apps in the shared storage do, which means all the system config is missing, which means Splunk doesn't work (runs on wrong port, can't authenticate etc). Is there a way to use both shared and non-shared apps at the same time with search head pooling? Cheers, Glenn ... View more