Is there any guidance on how and when to update the underlying OS for an AWS Splunk Enterprise Instance?

[n6BXGybt]

Splunk Enterprise Appliance in AWS Marketplace
Splunk Version: 6.3.0
Splunk Build: aa7d4b1ccb80

Is there guidance on how and when to update the underlying OS for the AWS Splunk Enterprise instance / appliance?

We have to run vulnerability scans and the appliance keeps popping up with vulnerabilities.

We want to update, but don't want to break the Splunk Enterprise install either.

[goodsellt · Accepted solution]

Generally depends on your company's policies I guess. In my case I generally plan on doing the last weekend of every month.

For things like a SH or SHC or a HF / UF, I generally just have a script that stops Splunk, performs the update command, and then starts Splunk or reboots the server if necessary.

On an index cluster it gets more complex, as you essentially need to follow the maintenance steps, which are put the cluster in maintenance mode, then "offline" a peer, perform the updates, then start Splunk or reboot the server, and once it's back alive and talking to the master then you do the next one (and so on). Once you're all finished you restore the cluster to standard operations and if you'd like with 6.5 you do a manual rebalance (since depending on how much time it takes your servers to go through the processes the data could be a bit unbalanced between them). I generally write scripts to do this and then monitor their progress in real time (though once you're comfortable with scripting you can write in error checking code to trigger alternate actions or alert you if something doesn't go to plan; plus you could use Splunk alerts to alert you to issues as well).

[n6BXGybt]

No reply so I simply use crontabs to update the security packages only:

yum update --security -y