Deployment Architecture

reload deploy-server causing splunk restart

keerthana_k
Communicator

Hi,

We have a distributed Splunk system installed and use deployment server to manage configurations. We have a python script which updates a few lookup CSV files and binary database files periodically. In the script, we run the reload deploy-server command to distribute the changed files across all the systems. Though the change is only lookup files, it is causing restart of splunk service at all the nodes. Is there anyway we can prevent the this restart? We have saved summary searches running and it is causing missing buckets of data.

Thanks in advance,
Keerthana

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi keerthana_k,
probably in your ServerClass you set for at least one of the Apps containing these lookups to restart Splunk, so when you launch "reload deploy-server" remote Splunks are restarted!
Bye.
Giuseppe

View solution in original post

0 Karma

koshyk
Super Champion

As per your post, it seems you are using "Deployment-server" to manage Search Head Cluster? if this is the case it is wrong. You should use "deployer" for the same.

Lookups normally don't tend to restart Splunk endpoints until you have forced the serverclass element of the server for restartSplunkd=true. If you want, you can make it restartSplunkd=false forcibly and have a go

0 Karma

keerthana_k
Communicator

I am not using a search head cluster. I just have two search heads which serve two different purposes. If I set restartSplunkd to false, then what will happen if I make any configuration change which might require splunk restart?

0 Karma

koshyk
Super Champion

you can put restartSplunkd to "false" for each app you push. So you can make it granular and package all your lookup into an app which you can say "false" for

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi keerthana_k,
probably in your ServerClass you set for at least one of the Apps containing these lookups to restart Splunk, so when you launch "reload deploy-server" remote Splunks are restarted!
Bye.
Giuseppe

0 Karma

keerthana_k
Communicator

We are setting restartSplunkd to true. So if I remove the configuration, then what will happen when I make other configuration changes that may require splunk restart?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi keerthana_k,
yes there's a problem!
I suggest to change the approach for lookups, two choices:

  • use your script to change them directly on all Search Heads instead on Deployment Server,
  • if possible, put all lookups in a different App, sharing them at Global level and deploying them without Splunk restart.

Bye.
Giuseppe

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Rounding off the Splunk Dashboard Contest

What does a contest-winning Splunk dashboard look like? In this case, it isn't in a browser tab at all. It ...

A Four Part Event Series: AI + Observability: AI Agents, LLMs, Apps, & Infrastructure

AI + Observability: AI Agents, LLMs, Apps, & Infrastructure The rapid evolution of artificial intelligence ...