Splunk Server OS Security patching

[maheshnc]

Hello,

Our operations team is supposed to perform OS Security patching on indexer cluster, search head, Heavy Forwarders, deployment server and licence master, I want to know, as a Splunk admin what are the prechecks and post-checks need to be performed? for example, do we need to take backup etc.

thanks in advance.

[kiran_panchavat]

@maheshnc 

In addition to the other steps, make sure to back up the /opt/splunk/bin directory on the Heavy Forwarders. If any custom scripts were placed directly in /opt/splunk/bin, it’s essential to include this directory in your pre-patching backup.

[PrewinThomas]

@maheshnc 

General best practices are,

Backups & Config Safety first
-Back up $SPLUNK_HOME/etc (configs, apps, knowledge objects).
-Back up critical KV Store collections
-VM snapshots or system‑level backups in case rollback is needed.

For Indexer Cluster:
-Put the cluster into maintenance mode.
-Better patch one peer at a time.

Post Checks
Service Validation
-Confirm Splunk service is running
-Verify web UI(Applicable ones) and CLI access

Cluster Health
-Indexer Cluster - all peers should be Up and In‑Sync.
-SHC - all members should be Up and Ready
-Disable maintenance mode once all peers are patched

Data Flow
-Perform simple searches (index=YOUR_INDEX | head 5) to confirm indexing is happening.
-Check forwarders are still connected (Settings → Forwarder Management)

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[maheshnc · Accepted solution]

Thanks!