We are having an internal debate concerning the frequency with which we should update our Splunk Enterprise software in our prod environment. I'm of a mind to do it roughly quarterly, which corresponds to Splunk's normal release cadence. Our admins prefer once per year. We're currently on 6.4.1.
I'm especially interested in what other large shops - with clustered indexers and search heads - are doing. I'll "vote up" every answer!
If you use orchestration software (such as Ansible) it makes upgrading much less of a headache. I'll do 90 systems about 2-3 times per year, and it takes around 2 hours to complete, with a few hours of prep work the first time around. Subsequent upgrades don't require any prep work outside of downloading the new Splunk package, and installing it on a test server for issues. I'm also running solo, so I don't have to coordinate with any sysadmins to get it done, just the end users when I do the SHs. Getting UFs upgraded is much more of an issue, since that involves the enterprise SCCM and *nix teams to be involved, and those upgrades often drag on for months, so they get done probably less than once per year.
I tend to update Splunk Enterprise once in 6 months in a normal scenario. But in case of emergency patch/security vulnerability we might update faster. Also I tend to go minor version 3 or above.. eg, 6.3.4,6.4.3, 6.4.4, 6.5.3, 6.5.4 etc..
as previous versions will contain fixes which may be real issue in large clustered systems. (eg: So though we have Splunk 6.6.0 available, I will still go with Splunk 6.5.4 as it is more stable for large environments)
Splunk UF's are more painful as we need to get approval from every single team one by one. But fortunatley, Splunk UF is backward compatible to a very long time. So unless there is a vulnerability we tend NOT to upgrade. Also some clients are Windows2008 are not supported by SplunkUF6.4.x. So it is more of a question, what you are going to achieve by upgrading Splunk UF as frequently as Enterprise
we also operate a clustered indexer and searchhead environment.
I am in the role as the splunk admin for the infrastructure as well as the application.
Our splunk environment runs on 6.4.1. And I would suggest upgrading once a year, because it means a lot of preparation and work.