topic Re: Splunk Server OS Security patching in Deployment Architecture

Splunk Server OS Security patching

maheshnc — Thu, 25 Sep 2025 06:46:23 GMT

Hello,

Our operations team is supposed to perform OS Security patching on indexer cluster, search head, Heavy Forwarders, deployment server and licence master, I want to know, as a Splunk admin what are the prechecks and post-checks need to be performed? for example, do we need to take backup etc.

thanks in advance.

Re: Splunk Server OS Security patching

PrewinThomas — Thu, 25 Sep 2025 07:03:57 GMT

@maheshnc

General best practices are,

Backups & Config Safety first
-Back up $SPLUNK_HOME/etc (configs, apps, knowledge objects).
-Back up critical KV Store collections
-VM snapshots or system‑level backups in case rollback is needed.

For Indexer Cluster:
-Put the cluster into maintenance mode.
-Better patch one peer at a time.

Post Checks
Service Validation
-Confirm Splunk service is running
-Verify web UI(Applicable ones) and CLI access

Cluster Health
-Indexer Cluster - all peers should be Up and In‑Sync.
-SHC - all members should be Up and Ready
-Disable maintenance mode once all peers are patched

Data Flow
-Perform simple searches (index=YOUR_INDEX | head 5) to confirm indexing is happening.
-Check forwarders are still connected (Settings → Forwarder Management)

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

Re: Splunk Server OS Security patching

kiran_panchavat — Thu, 25 Sep 2025 07:30:53 GMT

@maheshnc

In addition to the other steps, make sure to back up the /opt/splunk/bin directory on the Heavy Forwarders. If any custom scripts were placed directly in /opt/splunk/bin, it’s essential to include this directory in your pre-patching backup.

Re: Splunk Server OS Security patching

maheshnc — Thu, 25 Sep 2025 11:08:24 GMT

Thanks!