I'm hoping someone is able to help me find out what's going on with Splunk Stream and Netflow because I'm tearing my hair out trying to get it working.
I have a separate indexer and search head and am trying to use the independent stream forwarder. The forwarder host also has UF installed but not Splunk_TA_stream, incidentally I tried getting it working with the Splunk_TA_stream app and was also seeing similar results.
disabled = 0
port = 8088
dedicatedIoThreads = 8
description = Splunk Stream HEC
disabled = 0
index = main
token = <hec_token>
indexes = _internal,main
[splunk@<indexer> ~]$ netstat -antup | grep 8088
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 11580/splunkd
and running index=_internal sourcetype="stream:*" host="<forwarder>" gives me two sourcetypes, stream:log and stream:stats. stream:log gives me nothing of interest, just decode errors until the template is received, then these errors stop.
which suggests that netflow receivers are working as expected.
Running a tcpdump on the receiver host I can see that I am receiving genuine netflow v9 which is readable using wireshark.
I've looked at splunkd.log on the indexer and I'm not seeing anything that relates to the stream forwarder. I'm at a loss where to look next. I have gone through the documentation countless times over the last few days to make sure I'm not missing anything.