Custom Configuration File Not Replicating Across Search Head Cluster

[wipark]

Hi everyone,

I'm developing an app that uses a custom configuration file. I'm updating the file using the Splunk JavaScript SDK and REST API calls. In my lab setup (Splunk 9.4.1), everything works as expected—the custom config file is replicated correctly.

However, when I deploy the app in our production environment (also running Splunk 9.4.1), changes to the configuration file do not replicate to the other search heads. I used btool to verify that the file is not excluded from replication.

Has anyone encountered a similar issue? What steps can I take to investigate and debug this? What specific logs or configurations should I check?

[wipark]

I enabled additional logging on the production setup and updated the passwords.conf and customfile.conf files—first on the search head captain (sh01), and then on another member (sh03).

In both cases, logs were generated for the passwords.conf updates. However, there were no logs related to the customfile.conf file.

The first set of logs corresponds to the update on the captain (sh01), and the second set corresponds to the update on the member (sh03). Sensitive fields have been redacted or anonymized.

05-30-2025 10:10:10.185 +0000 DEBUG ConfReplication [1692624 TcpChannelThread] - addCommit: to_repo=https://sh01.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=
05-30-2025 10:10:13.591 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - pullFrom_Locked: status=handling, from_repo=https://sh01.acme.com:8089, to_repo=https://sh03.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=
05-30-2025 10:10:13.591 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - pullFrom_Locked: status=applied, reason="", from_repo=https://sh01.acme.com:8089, to_repo=https://sh03.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599813, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=

05-30-2025 10:10:10.497 +0000 DEBUG ConfReplication [3612371 TcpChannelThread] - addCommit: to_repo=https://sh03.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=
05-30-2025 10:10:10.501 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - ConfOpStorage: toPush ptr=0x7ff55ebfcd50, pos=0, repo=https://sh03.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=
05-30-2025 10:10:10.507 +0000 DEBUG ConfReplication [1993289 TcpChannelThread] - acceptPush_Locked: status=handling, from_repo=https://sh03.acme.com:8089, to_repo=https://sh01.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=
05-30-2025 10:10:10.511 +0000 DEBUG ConfReplication [1993289 TcpChannelThread] - acceptPush_Locked: status=applied, reason="", from_repo=https://sh03.acme.com:8089, to_repo=https://sh01.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ {  }, removable: yes ]\n }, extra_payload=

[wipark]

Sorry, sent this twice and was not able to delete. 

[wipark]

I’m not sure if this helps, but changes made on the search head cluster captain do appear in the bundle under /opt/splunk/var/run.

[Prewin27]

@wipark 
Check for Replication Quarantine or Bundle Issues
Large or problematic files (e.g., big CSV lookups) can cause replication to fail or be quarantined.

Review the metrics.log and splunkd.log on all SHC members for replication errors or warnings

Test Manual Change
Make a simple change to a standard file (e.g. props.conf) via the UI or REST API and see if it replicates.

If standard files replicate but your custom file does not, it’s likely a file location or inclusion issue.

If the cluster is out of sync - Force Resync if required eg: splunk resync shcluster-replicated-config

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos. Thanks!

[wipark]

@Prewin27 , I have reviewed the app configuration and logs but could not find any errors related to the issue.
The application includes a passwords.conf file, which is replicated immediately across the SHC nodes. I wasn't able to find any exclusions regarding the custom file. 

[livehybrid]

Hi @wipark 

Within your app, have you set a conf_replication_include key/value pair to tell the system to replicate it?

conf_replication_include.<conf_file_name> = <boolean>

e.g.

conf_replication_include.yourCustomConfFile = true

Is your production environment a different architecture (e.g. SHC vs single instance) than your local environment?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[wipark]

---

Hi @livehybrid 

Within your app, have you set a conf_replication_include key/value pair to tell the system to replicate it?

conf_replication_include.<conf_file_name> = <boolean>

---

Yes I have set that. 

---

Is your production environment a different architecture (e.g. SHC vs single instance) than your local environment?

---

No, both are SHCs.

[livehybrid]

Hmm that is odd. It might be worth checking for any custom distsearch.conf settings on your production environment which might be blocking things. Please can you do a btool against distsearch and look for anything which is in local?

$SPLUNK_HOME/bin/splunk cmd btool distsearch list --debug

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[wipark]

I copied the effective distsearch.conf from production (using btool) to my lab setup under $SPLUNK_HOME/etc/system/local. After restarting Splunk, I verified it again with btool to confirm it matched the production configuration. Replication is still working fine in the lab setup, so it seems there's nothing wrong with distsearch.conf

[wipark]

I found that only a couple of lookups were referenced in local files. I also reviewed all allowlist and denylist settings in distsearch.conf, and everything appears to be in order.

Additionally, I compared the server.conf files in both environments using btool. There were no significant differences aside from some expected variations in names and IDs. The only notable difference—though I doesn't seem to be relevant—is that captain_is_adhoc_searchhead is set to true in production.