We tried to remove the default group as you suggested but it gave us the same error. We don't have to send data to another Splunk, on the other side there will be Fluentd that will capture the data. At the moment we are trying to send data to a socket opened with netcat on another device in the same subnet. We see coming data on netcat, but Splunk crashes on the indexers. This is the btool output related to output.conf: /opt/splunk/etc/system/local/outputs.conf [tcpout] /opt/splunk/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30 /opt/splunk/etc/system/default/outputs.conf autoLBFrequency = 30 /opt/splunk/etc/system/default/outputs.conf autoLBVolume = 0 /opt/splunk/etc/system/default/outputs.conf blockOnCloning = true /opt/splunk/etc/system/default/outputs.conf blockWarnThreshold = 100 /opt/splunk/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256 /opt/splunk/etc/system/default/outputs.conf compressed = false /opt/splunk/etc/system/default/outputs.conf connectionTTL = 0 /opt/splunk/etc/system/default/outputs.conf connectionTimeout = 20 /opt/splunk/etc/system/default/outputs.conf disabled = false /opt/splunk/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5 /opt/splunk/etc/system/default/outputs.conf dropEventsOnQueueFull = -1 /opt/splunk/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1 /opt/splunk/etc/system/default/outputs.conf enableOldS2SProtocol = false /opt/splunk/etc/system/default/outputs.conf forceTimebasedAutoLB = false /opt/splunk/etc/system/default/outputs.conf forwardedindex.0.whitelist = .* /opt/splunk/etc/system/default/outputs.conf forwardedindex.1.blacklist = _.* /opt/splunk/etc/system/default/outputs.conf forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup|_configtracker) /opt/splunk/etc/system/local/outputs.conf forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection) /opt/splunk/etc/system/default/outputs.conf forwardedindex.filter.disable = false /opt/splunk/etc/system/default/outputs.conf heartbeatFrequency = 30 /opt/splunk/etc/system/local/outputs.conf indexAndForward = true /opt/splunk/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2 /opt/splunk/etc/system/default/outputs.conf maxFailuresPerInterval = 2 /opt/splunk/etc/system/default/outputs.conf maxQueueSize = auto /opt/splunk/etc/system/default/outputs.conf readTimeout = 300 /opt/splunk/etc/system/default/outputs.conf secsInFailureInterval = 1 /opt/splunk/etc/system/default/outputs.conf sendCookedData = true /opt/splunk/etc/system/default/outputs.conf sslQuietShutdown = false /opt/splunk/etc/system/default/outputs.conf sslVersions = tls1.2 /opt/splunk/etc/system/default/outputs.conf tcpSendBufSz = 0 /opt/splunk/etc/system/default/outputs.conf useACK = false /opt/splunk/etc/system/default/outputs.conf useClientSSLCompression = true /opt/splunk/etc/system/default/outputs.conf writeTimeout = 300 /opt/splunk/etc/system/local/outputs.conf [tcpout:external_system] /opt/splunk/etc/system/local/outputs.conf disabled = false /opt/splunk/etc/system/local/outputs.conf sendCookedData = false /opt/splunk/etc/system/local/outputs.conf server = <external_server>:<external_port>   ... View more

We followed current documentation: [tcpout] defaultGroup = <comma-separated list> * A comma-separated list of one or more target group names, specified later in [tcpout:<target_group>] stanzas. * The forwarder sends all data to the specified groups. * If you don't want to forward data automatically, don't configure this setting. * Can be overridden by the '_TCP_ROUTING' setting in the inputs.conf file, which in turn can be overridden by a props.conf or transforms.conf modifier. * Starting with version 4.2, this setting is no longer required. Data forwarding is working, but the state of the cluster is invalid. We also noted these crash logs and we think they are related to this problem: Received fatal signal 6 (Aborted) on PID 2521742. Cause: Signal sent by PID 2521742 running under UID 1001. Crashing thread: TcpOutEloop ........... Backtrace (PIC build): [0x00007F02A9F91A7C] pthread_kill + 300 (libc.so.6 + 0x6EA7C) [0x00007F02A9F3D476] raise + 22 (libc.so.6 + 0x1A476) [0x00007F02A9F237F3] abort + 211 (libc.so.6 + 0x7F3) [0x0000556B5A5B0FA9] ? (splunkd + 0x1A52FA9) [0x0000556B5BA12B6E] _ZN11TimeoutHeap18runExpiredTimeoutsER13MonotonicTime + 670 (splunkd + 0x2EB4B6E) [0x0000556B5B939260] _ZN9EventLoop18runExpiredTimeoutsER13MonotonicTime + 32 (splunkd + 0x2DDB260) [0x0000556B5B93A690] _ZN9EventLoop3runEv + 208 (splunkd + 0x2DDC690) [0x0000556B5A97185E] _ZN11Distributed11EloopRunner4mainEv + 206 (splunkd + 0x1E1385E) [0x0000556B5BA0957D] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2EAB57D) [0x0000556B5BA0A413] _ZN6Thread8callMainEPv + 147 (splunkd + 0x2EAC413) [0x00007F02A9F8FB43] ? (libc.so.6 + 0x6CB43) [0x00007F02AA021A00] ? (libc.so.6 + 0xFEA00) Linux / splunk-indexer01 / 5.15.0-76-generic / #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 / x86_64 assertion_failure="!_current_timeout_was_readded" assertion_function="void TimeoutHeap::assert_didnt_get_readded() const" assertion_file="/builds/splcore/main/src/util/TimeoutHeap.h:527" /etc/debian_version: bookworm/sid Last errno: 0 Threads running: 85 Runtime: 61.996351s argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd] Regex JIT enabled .......   ... View more

Hi, thanks for your reply. We tried this approach but we had the problem described in the previous answer. Maybe they are related to the remote queue size as you said. Is there a way to control the remote queue size or length in tcpout mode?   ... View more

We didn't put indexAndForward under the [tcpout] because the documentation says:   * This setting is only available for heavy forwarders.   But we also tried with this configuration and it didn't work the same:   [tcpout] indexAndForward = true defaultGroup=external_system forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection) [tcpout:external_system] disabled=false sendCookedData=false server=<external_host>:<external_port>   We applied this config by bundle push on the indexers. The main issue is that the restart never ends, as you can see from the attached picture. At least one indexer remains in a "pending" state. After apply this config, search factor and replication factor cannot be met and ALL the indexes were not fully searchable. Despite of the invalide state of the cluster, we saw coming data on the external system.   ... View more

Thanks for your response. We tried to distribute this configuration on our indexes, but it didn't work. We saw coming data on the external system, but Splunk became not searchable and replication factor was not met. Do you see something wrong with this one? [indexAndForward] index=true selectiveIndexing=false [tcpout] defaultGroup=external_system forwardedindex.3.blacklist = (_internal|_audit|_telemetry|_introspection) [external_system] indexAndForward = true [tcpout:external_system] disabled=false sendCookedData=false server=<external_system>:<external_port>   ... View more

Thanks both! ... View more