We finally got stream working - but more of a work around.  The problem is in part due to starting the UF using systemd, which allocates CPU slices for different processes.   When using systemd to start the UF, stream fails.   Disabling start on boot, and manually starting the UF from ./slunk start, stream works. The second part is that when the UF starts, ownership of all the UF files is chowned  splunk:splunk.  This seems logical to ensure the UF runs as splunk (or splunkfwd).  However, when stream is initially installed, the set_permissions.sh changes ownership of ../Splunk_TA_stream/Linux_x86_64/streamfwd-rhel6 to root.  Starting the UF undoes this, changing ownership back to splunk.   We made streamfwwd-rhel6 immutable - which did prevent the ownership change back to splunk, but stream still failed when starting with systemd. Ultimately, we had to disable systemd, make streamfwd-rhel6 immutable (after running set_permissions.sh), then start the UF manually via /splunk start.     Splunk needs to fix this so stream works as expected without having to disable boot-start and set the immutable flag.

Hi @gcusello  No, the 'Splunk_TA_f5-bigip' app is on the DS but not the IDXs/CM. Is that something that ought to be pushed out to the CM/IDX?  We have a local app that is specific to our program with a ruleset for that f5:bigip:syslog but that is just specifying the hosts that route data to that index.  How can I check via the regex what sourcetypes the data has?  Thank you!   

Hi Splunkers I notice the same issue and wonder really why Splunk is not fixing this issue? Is seems to be an incompatibility on the VMware stack with the streamfwd service.  I use Splunk Universalforwarder 9.1.2 and Splunk Stream 9.1.1. Specially the installation on Universalforwarders fails massively on Linux systems which makes Splunk Stream not really usable in a distributed environment with Linux systems. My streamfwd.log tells always the same error:   2024-03-08 14:59:54 INFO [139974317471680] (CaptureServer.cpp:2001) stream.CaptureServer - Starting data capture 2024-03-08 14:59:54 INFO [139974317471680] (SnifferReactor/SnifferReactor.cpp:161) stream.SnifferReactor - Starting network capture: sniffer 2024-03-08 14:59:54 ERROR [139974317471680] (SnifferReactor/PcapNetworkCapture.cpp:238) stream.NetworkCapture - SnifferReactor unrecognized link layer for device <eth0>: 253 2024-03-08 14:59:54 FATAL [139974317471680] (CaptureServer.cpp:2337) stream.CaptureServer - SnifferReactor was unable to start packet capturesniffer 2024-03-08 14:59:54 INFO [139974317471680] (CaptureServer.cpp:2362) stream.CaptureServer - Done pinging stream senders (config was updated) 2024-03-08 14:59:54 INFO [139974317471680] (main.cpp:1109) stream.main - streamfwd has started successfully (version 8.1.1 build afdcef4b) 2024-03-08 14:59:54 INFO [139974317471680] (main.cpp:1111) stream.main - web interface listening on port 8889       As you all can see, my streamfwd.conf is more or less the same as all of you have also. No matter if for example i change the ipAddr to 0.0.0.0. I always get the same error.     [streamfwd] logConfig = streamfwdlog.conf port = 8889 ipAddr = 127.0.0.1 ## --> Token HFWD httpEventCollectorToken = ba4a2b2-2544-55e3-22ft-234vt68m0szp ## --> Specify the interface streamfwdcapture.1.interface = eth0       Side remark: If I reinstall Splunk Enterprise 9.1.2 on the same server on which UniversalForwarder 9.1.2 with Splunk Stream 9.1.1 was installed, Splunk Stream works. That sounds like a bug in Splunk_TA_stream. Would be great to hear a statement of Splunk within the next weeks. Kind regards Patrick  

Got it working after adding src_ip field. Splunk is a long journey to learning basic stuff source="udp:514" index="syslog" NOT src_ip IN (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) action=DROP  

Hi @DanAlexander , looks like this is not supported at the moment : https://docs.splunk.com/Documentation/AddonBuilder/4.1.4/UserGuide/Installation "Add-on Builder is not supported in a search head cluster or index cluster environment."   You can upvote this idea : https://ideas.splunk.com/ideas/APPSID-I-843      

I described in OP that field_A and field_B share the value. So there's nothing I can do to join the rest of the corresponding fields together? Yes, I did find evidence of field_B in source_2 having a value of 11111179 and a non-null value of field_C, which is what I intended to refer to in my previous post.  In regards to the mock code,  the search worked, so I'm guessing the flags were ignored. Either way, if it is indeed a data problem, I'm not seeing it in my logs. Thank you for the insight though. Will update this post if that does end up being the problem.

Hi @molko13, usually few persons are enabled to open a Case to Splunk Support for each entitlement, and usually is someone of the customer to open cases. I'm enabled to open Cases for some selected customers that enabled me to do this, but this is an exception. If you ned to do this, you must ask to your customer to ask to Splunk to enable you, otherwise isn't possible. Ciao. Giuseppe

Hi @anandhalagaras1, you should install in your Splunk the Splunk Dashboard Examples App (https://splunkbase.splunk.com/app/1603). One of the examples is just what you're searching: the In page Drilldown. Ciao. Giuseppe