Splunk Stream not showing netflow data

[neo_traffic]

I installed Splunk Stream per the instructions and I see data coming in when I run a search sourcetype=stream:netflow.

In the Stream App, I only see the local data, nothing from my netflow devices.

I am running it as a standalone server.

My configs are as follows:

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf

[streamfwd]
logConfig = streamfwdlog.conf
port = 8889

netflowReceiver.0.ip = XXX.XXX.XXX.XXX (real IP hidden)
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow
netflowReceiver.0.protocol = udp
netflowReceiver.0.decodingThreads = 4

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf

[streamfwd://streamfwd]
splunk_stream_app_location = http://localhost:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf

[http]
disabled = 0
port = 8088
dedicatedIoThreads = 8

[http://streamfwd]
disabled = 0
index=main
token = dcb7872a-9438-4e2e-a314-a20d2991df7b
indexes=_internal,main

netstat -l shows me:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN
tcp 0 0 localhost:8065 0.0.0.0:* LISTEN
tcp 0 0 localhost:domain 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:omniorb 0.0.0.0:* LISTEN
tcp 0 0 localhost:8889 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8191 0.0.0.0:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
udp 0 0 neo-monitor:9995 0.0.0.0:*
udp 0 0 localhost:domain 0.0.0.0:*
raw6 0 0 [::]:ipv6-icmp [::]:* 7
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] SEQPACKET LISTENING 12213 /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 307858 /run/user/0/systemd/private
unix 2 [ ACC ] STREAM LISTENING 307864 /run/user/0/gnupg/S.gpg-agent
unix 2 [ ACC ] STREAM LISTENING 307865 /run/user/0/gnupg/S.gpg-agent.ssh
unix 2 [ ACC ] STREAM LISTENING 307866 /run/user/0/gnupg/S.dirmngr
unix 2 [ ACC ] STREAM LISTENING 307867 /run/user/0/gnupg/S.gpg-agent.browser
unix 2 [ ACC ] STREAM LISTENING 307868 /run/user/0/gnupg/S.gpg-agent.extra
unix 2 [ ACC ] STREAM LISTENING 11797 /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 11804 /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 11909 /run/lvm/lvmpolld.socket
unix 2 [ ACC ] STREAM LISTENING 11950 /run/lvm/lvmetad.socket
unix 2 [ ACC ] STREAM LISTENING 16749 /run/snapd.socket
unix 2 [ ACC ] STREAM LISTENING 16737 /var/snap/lxd/common/lxd/unix.socket
unix 2 [ ACC ] STREAM LISTENING 16770 @ISCSIADM_ABSTRACT_NAMESPACE
unix 2 [ ACC ] STREAM LISTENING 16742 /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 16751 /run/snapd-snap.socket
unix 2 [ ACC ] STREAM LISTENING 16766 /run/uuidd/request
unix 2 [ ACC ] STREAM LISTENING 16768 /run/acpid.socket

streamfwd.log shows me:

2019-04-04 15:01:47 INFO 140379561435840 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/data
2019-04-04 15:01:47 INFO 140379561435840 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/ui
2019-04-04 15:01:48 INFO 140379561435840 stream.CaptureServer - Default configuration directory: /opt/splunk/etc/apps/Splunk_TA_stream/default
2019-04-04 15:01:48 ERROR 140379561435840 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:01:48 INFO 140379561435840 stream.main - streamfwd has started successfully (version 7.1.2 build 157)
2019-04-04 15:01:48 INFO 140379561435840 stream.main - web interface listening on port 8889
2019-04-04 15:01:54 ERROR 140379406546688 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:01:59 ERROR 140379406546688 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:02:04 ERROR 140379423332096 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:02:09 ERROR 140379406546688 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:02:14 ERROR 140379406546688 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:02:19 ERROR 140379406546688 stream.CaptureServer - Unable to ping server (c7a112ef-d447-423b-8231-e5a9206b86aa): Unable to establish connection to localhost: Connection refused
2019-04-04 15:02:26 INFO 140379406546688 stream.CaptureServer - Netflow receiver configuration defined; disabling default automatic promiscuous mode packet capture on all available interfaces. Configure one or more streamfwdcapture parameters in streamfwd.conf to enable network packet capture.
2019-04-04 15:02:26 INFO 140379406546688 stream.SnifferReactor - No packet processors configured
2019-04-04 15:02:26 INFO 140379406546688 stream.CaptureServer - Starting data capture
2019-04-04 15:02:26 INFO 140379406546688 stream.SnifferReactor - Starting network capture: sniffer

I am running Ubuntu 18.04, Splunk 7.2.5.1, Splunk Stream 7.1.2

Any help would be appreciated.

[michaeljorgense]

This looks like close something I am experiencing. My understanding is the streamfwd binary needs to phone home to the Splunk App for Stream as described here:

https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/DeploymentArchitecture#How_str...

This is where you configure streamfwd to talk to the Stream App:

https://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/ConfigureStreamForwarder#Verif...

In your config this is set to:

splunk_stream_app_location = http://localhost:8000/en-us/custom/splunk_app_stream/

To me it looks like your logs are showing that streamfwd is getting a connection refused when connecting via http to localhost on tcp port 8000. It's getting a connection refused when attempting that.

Are you able to, on that same splunk server, access:

http://localhost:8000/en-us/custom/splunk_app_stream/ping

If you can't that might indicate your problem, i.e. a local firewall, DNS resolution of "localhost" etc might not be working for you?