This is also an issue for me (not using aggregations). All the $trellis...$ tokens don't work when passing to a custom search. My workaround was to copy the URI generated for my search, and insert the $trellis...$ token in the proper place (I used a |u for URL encoding but not sure it's necessary). When using the "Link to Custom URL" drilldown, the tokens work just fine. Downside is that now the user gets the  "Redirecting Away From Splunk" message prior to being redirected.

Hi Have you checked that your Windows version is supported by Splunk 9.1.1?  As you could see from https://docs.splunk.com/Documentation/Splunk/9.1.1/ReleaseNotes/Deprecatedfeatures#Removed_features_in_version_9.1 Windows 2016 has removed from supported OS version. r. Ismo

You should look what you need to put on UF and HF vs. what is needed on indexer in outputs.conf. Those are different thing as normally indexers just write events to disks. On props.conf is this one for indexing and clone events to another destination. As you could see there is no default group definition. # Clone events to groups indexer1 and indexer2. Also, index all this data # locally as well. [tcpout] indexAndForward=true [tcpout:indexer1] server=Y.Y.Y.Y:9997 [tcpout:indexer2] server=X.X.X.X:6666 I suppose than when you set default group here it just changing this behaviour somehow and then it cannot store events inside this cluster. Seems to be some kind of timeout which happened before that crash. Have you see any events on target system? Based on port I assume that target is also splunk? If so, you should remove "sendCookedData = false" to send S2S data to remote. My guess is that this should work [tcpout] indexAndForward=true [tcpout:external_system] disabled=false forwardedindex.0.blacklist = (_internal|_audit|_telemetry|_introspection) server=<external_system>:9997  

Thank you @richgalloway and @bowesmana  - I'd accept both as the solution if I could as I learned about the return and format commands from you both. I accepted return as the solution since I wanted to use the IN search, and couldn't format the format command to remove the column names from the generated string. Not sure this is right, but I ended up having to use an eval command to append quotesa and commas to my values, prior to the return statement. In the end, it was something like...  index=syslog src_ip IN ( [ | tstats count from datamodel=Random by ips | stats values(ips) as IP | eval IP = "\"".IP."\"," | return $IP ] )  Thanks again!

We followed current documentation: [tcpout] defaultGroup = <comma-separated list> * A comma-separated list of one or more target group names, specified later in [tcpout:<target_group>] stanzas. * The forwarder sends all data to the specified groups. * If you don't want to forward data automatically, don't configure this setting. * Can be overridden by the '_TCP_ROUTING' setting in the inputs.conf file, which in turn can be overridden by a props.conf or transforms.conf modifier. * Starting with version 4.2, this setting is no longer required. Data forwarding is working, but the state of the cluster is invalid. We also noted these crash logs and we think they are related to this problem: Received fatal signal 6 (Aborted) on PID 2521742. Cause: Signal sent by PID 2521742 running under UID 1001. Crashing thread: TcpOutEloop ........... Backtrace (PIC build): [0x00007F02A9F91A7C] pthread_kill + 300 (libc.so.6 + 0x6EA7C) [0x00007F02A9F3D476] raise + 22 (libc.so.6 + 0x1A476) [0x00007F02A9F237F3] abort + 211 (libc.so.6 + 0x7F3) [0x0000556B5A5B0FA9] ? (splunkd + 0x1A52FA9) [0x0000556B5BA12B6E] _ZN11TimeoutHeap18runExpiredTimeoutsER13MonotonicTime + 670 (splunkd + 0x2EB4B6E) [0x0000556B5B939260] _ZN9EventLoop18runExpiredTimeoutsER13MonotonicTime + 32 (splunkd + 0x2DDB260) [0x0000556B5B93A690] _ZN9EventLoop3runEv + 208 (splunkd + 0x2DDC690) [0x0000556B5A97185E] _ZN11Distributed11EloopRunner4mainEv + 206 (splunkd + 0x1E1385E) [0x0000556B5BA0957D] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2EAB57D) [0x0000556B5BA0A413] _ZN6Thread8callMainEPv + 147 (splunkd + 0x2EAC413) [0x00007F02A9F8FB43] ? (libc.so.6 + 0x6CB43) [0x00007F02AA021A00] ? (libc.so.6 + 0xFEA00) Linux / splunk-indexer01 / 5.15.0-76-generic / #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 / x86_64 assertion_failure="!_current_timeout_was_readded" assertion_function="void TimeoutHeap::assert_didnt_get_readded() const" assertion_file="/builds/splcore/main/src/util/TimeoutHeap.h:527" /etc/debian_version: bookworm/sid Last errno: 0 Threads running: 85 Runtime: 61.996351s argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd] Regex JIT enabled .......  

Hi here is couple of same kind of cases:  https://community.splunk.com/t5/Dashboards-Visualizations/Why-am-I-Receiving-quot-A-custom-JavaScript-error-caused-an/m-p/619270 https://community.splunk.com/t5/Dashboards-Visualizations/Receiving-quot-A-custom-JavaScript-error-caused-an-issue-loading/m-p/643743 https://community.splunk.com/t5/Dashboards-Visualizations/How-do-I-fix-this-custom-javascript-Error-Notification-pop-up-on/m-p/648232 Maybe this was already fixed on the latest versions? r. Ismo

Hi, thanks for your reply. We tried this approach but we had the problem described in the previous answer. Maybe they are related to the remote queue size as you said. Is there a way to control the remote queue size or length in tcpout mode?