Deployment Architecture

Splunk Server OS Security patching

maheshnc
Path Finder

Hello,

Our operations team is supposed to perform OS Security patching on indexer cluster, search head, Heavy Forwarders, deployment server and licence master, I want to know, as a Splunk admin what are the prechecks and post-checks need to be performed? for example, do we need to take backup etc.

thanks in advance.

0 Karma
1 Solution

maheshnc
Path Finder
0 Karma

kiran_panchavat
SplunkTrust
SplunkTrust

@maheshnc 

In addition to the other steps, make sure to back up the /opt/splunk/bin directory on the Heavy Forwarders. If any custom scripts were placed directly in /opt/splunk/bin, it’s essential to include this directory in your pre-patching backup.

Did this help? If yes, please consider giving kudos, marking it as the solution, or commenting for clarification — your feedback keeps the community going!
0 Karma

PrewinThomas
Motivator

@maheshnc 

General best practices are,

Backups & Config Safety first
-Back up $SPLUNK_HOME/etc (configs, apps, knowledge objects).
-Back up critical KV Store collections
-VM snapshots or system‑level backups in case rollback is needed.

For Indexer Cluster:
-Put the cluster into maintenance mode.
-Better patch one peer at a time.

Post Checks
Service Validation
-Confirm Splunk service is running
-Verify web UI(Applicable ones) and CLI access

Cluster Health
-Indexer Cluster - all peers should be Up and In‑Sync.
-SHC - all members should be Up and Ready
-Disable maintenance mode once all peers are patched


Data Flow
-Perform simple searches (index=YOUR_INDEX | head 5) to confirm indexing is happening.
-Check forwarders are still connected (Settings → Forwarder Management)

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

0 Karma

maheshnc
Path Finder

Thanks!

0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...