I enabled additional logging on the production setup and updated the passwords.conf and customfile.conf files—first on the search head captain (sh01), and then on another member (sh03). In both cases, logs were generated for the passwords.conf updates. However, there were no logs related to the customfile.conf file. The first set of logs corresponds to the update on the captain (sh01), and the second set corresponds to the update on the member (sh03). Sensitive fields have been redacted or anonymized. 05-30-2025 10:10:10.185 +0000 DEBUG ConfReplication [1692624 TcpChannelThread] - addCommit: to_repo=https://sh01.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:13.591 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - pullFrom_Locked: status=handling, from_repo=https://sh01.acme.com:8089, to_repo=https://sh03.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:13.591 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - pullFrom_Locked: status=applied, reason="", from_repo=https://sh01.acme.com:8089, to_repo=https://sh03.acme.com:8089, op_id=1252dcef9d0f33386e7feab562eba92d424515ea, applied_at=1748599813, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:10.497 +0000 DEBUG ConfReplication [3612371 TcpChannelThread] - addCommit: to_repo=https://sh03.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:10.501 +0000 DEBUG ConfReplication [2010047 ConfReplicationThread] - ConfOpStorage: toPush ptr=0x7ff55ebfcd50, pos=0, repo=https://sh03.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:10.507 +0000 DEBUG ConfReplication [1993289 TcpChannelThread] - acceptPush_Locked: status=handling, from_repo=https://sh03.acme.com:8089, to_repo=https://sh01.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload= 05-30-2025 10:10:10.511 +0000 DEBUG ConfReplication [1993289 TcpChannelThread] - acceptPush_Locked: status=applied, reason="", from_repo=https://sh03.acme.com:8089, to_repo=https://sh01.acme.com:8089, op_id=481af55d46acfb6f4da973c3aac4af9e8ab2e0e6, applied_at=1748599810, asset_id=c922db4bf111d426f1e8eb78181cb8f43b185f52, asset_uri=/nobody/custom-app/passwords/credential:custom-app_realm:password:, optype=WRITE_STANZA, payload={ password = REDACTED [ { }, removable: yes ]\n }, extra_payload=   ... View more

Sorry, sent this twice and was not able to delete.  ... View more

I copied the effective distsearch.conf from production (using btool) to my lab setup under $SPLUNK_HOME/etc/system/local. After restarting Splunk, I verified it again with btool to confirm it matched the production configuration. Replication is still working fine in the lab setup, so it seems there's nothing wrong with distsearch.conf ... View more

I’m not sure if this helps, but changes made on the search head cluster captain do appear in the bundle under /opt/splunk/var/run. ... View more

I found that only a couple of lookups were referenced in local files. I also reviewed all allowlist and denylist settings in distsearch.conf, and everything appears to be in order. Additionally, I compared the server.conf files in both environments using btool. There were no significant differences aside from some expected variations in names and IDs. The only notable difference—though I doesn't seem to be relevant—is that captain_is_adhoc_searchhead is set to true in production. ... View more

@PrewinThomas , I have reviewed the app configuration and logs but could not find any errors related to the issue. The application includes a passwords.conf file, which is replicated immediately across the SHC nodes. I wasn't able to find any exclusions regarding the custom file.  ... View more

Hi @livehybrid  Within your app, have you set a conf_replication_include key/value pair to tell the system to replicate it? conf_replication_include.<conf_file_name> = <boolean>   Yes I have set that.  Is your production environment a different architecture (e.g. SHC vs single instance) than your local environment?   No, both are SHCs. ... View more

Upon further investigation It seems the _time field needs to be present for Splunk to show the results.  A code like this works: def stream(self, events): yield {"myfield": "fff", "_time": "1748073052.114"} ... View more

Thanks, that solved the problem. It seems that for the code to work correctly, I need to yield the original record, or at least some fields of it. I haven’t tested thoroughly yet, but today I started testing based on this example.  The following code works as expected for both of these queries: index=_internal | head 1 | table host | streamingcsc index=_internal | head 1 | streamingcsc #!/usr/bin/env python import os, sys sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib")) from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators @Configuration() class StreamingCSC(StreamingCommand): def stream(self, records): for record in records: record["event"] = str(type(records)) yield record dispatch(StreamingCSC, sys.argv, sys.stdin, sys.stdout, __name__)   But The following code only works with this query: index=_internal | head 1 | table host | streamingcsc #!/usr/bin/env python import os, sys sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib")) from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators @Configuration() class StreamingCSC(StreamingCommand): def stream(self, records): for record in records: yield {"event": str(type(records))} dispatch(StreamingCSC, sys.argv, sys.stdin, sys.stdout, __name__)   ... View more

This is a simple code to reproduce the problem. My actual code iterates over the events. The problem is when I dont use the table command before my customcommand there is no events to iterate over. In this case without table there is no results not even one. Also my search in this example uses head 1, thus there is only one input  result.  I will try your version when I got back to work. ... View more

This is a simple code to reproduce the problem. My actual code iterates over the events. The problem is when I dont use the table command before my customcommand there is no events to iterate over. In this case without table there is no results not even one. Also my search in this example uses head 1, thus there is only one input  result. ... View more

I am not particularly concerned about the vulnerability right now. But this old OpenSSL version is causing problems when I try to develop new apps. I know it might not be a Splunk problem. urllib3 v2 is a dependency of a package that I need to use. As I understand this version of OpenSSL is not supported by the library or newer version of Python. The error message is:     urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2zi-fips 1 Aug 2023'. See: https://github.com/urllib3/urllib3/issues/2168     ... View more