topic Re: Stream app - Netflowreceiver problem in All Apps and Add-ons

Stream app - Netflowreceiver problem

heskez — Wed, 14 Mar 2018 09:10:42 GMT

Hi There, I've configured the stream app and streamfwd.log as follow:

netflowReceiver.0.ip = 192.168.1.2
netflowReceiver.0.port = 9996
netflowReceiver.0.protocol = udp
netflowReceiver.0.decoder = netflow

UDP Netflow is coming in on the splunk server, confirmed with TCPDUMP

However, I don't get the netflow data and see these kinds of errors in streamfwd.log:

Caught exception in openDatagramListenersystem:99 bind
Unable to start any Netflow Receivers

Re: Stream app - Netflowreceiver problem

p_gurav — Wed, 14 Mar 2018 12:34:08 GMT

Can you provide inputs.conf ?

Re: Stream app - Netflowreceiver problem

heskez — Tue, 29 Sep 2020 18:26:46 GMT

Local:

[streamfwd://streamfwd]
splunk_stream_app_location = http://localhost:8000/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

Default:

[streamfwd]
disabled = true

Re: Stream app - Netflowreceiver problem

heskez — Thu, 15 Mar 2018 17:11:16 GMT

Was this inputs.conf sufficient in order to proceed?

Re: Stream app - Netflowreceiver problem

heskez — Thu, 15 Mar 2018 17:12:20 GMT

Is this inputs.conf sufficient to proceed?

Re: Stream app - Netflowreceiver problem

heskez — Fri, 16 Mar 2018 14:45:48 GMT

Hello there?

Re: Stream app - Netflowreceiver problem

heskez — Mon, 19 Mar 2018 16:08:11 GMT

Anyone out there?

Re: Stream app - Netflowreceiver problem

gjanders — Wed, 21 Mar 2018 08:17:45 GMT

Is this an independent stream forwarder and are you running the process as root? Or as a non-root user?
Also you have a Splunk instance with the webgui enabled and the stream application on the same host?

Re: Stream app - Netflowreceiver problem

heskez — Wed, 21 Mar 2018 08:28:15 GMT

Yes, Everything, the Splunk instance with the web gui, the stream application is installed on one host.
The process runs as root.

Re: Stream app - Netflowreceiver problem

gjanders — Wed, 21 Mar 2018 10:58:16 GMT

Ok so the IP 192.168.1.2 is configured on the server?
Does netstat -an | grep 9996 show anything listening on that port?

I do not see netflowReceiver.0.protocol = udp as a mentioned line in Configure netflow collector but not sure if that is making any difference here (netflow is normally udp so you should be able to drop this line).

Finally when you said:
"Hi There, I've configured the stream app and streamfwd.log as follow: ", I assume you mean streamfwd.conf ?

Re: Stream app - Netflowreceiver problem

heskez — Wed, 21 Mar 2018 11:28:01 GMT

The 192.168.1.2 is the ip address of the firewall which is sending the netflow information to Splunk machine.

Output on splunk: udp 0 0 0.0.0.0:9996 0.0.0.0:*

Yes offcourse streamfwd.conf 🙂

Re: Stream app - Netflowreceiver problem

gjanders — Wed, 21 Mar 2018 21:08:41 GMT

netflowReceiver.0.ip = should be the IP of the host listening for the data (i.e. your server), not the IP of the server sending the data...

Re: Stream app - Netflowreceiver problem

gjanders — Wed, 21 Mar 2018 21:28:10 GMT

Also can you run a netstat -anp | grep 9996
That should show you what process is using 9996

Re: Stream app - Netflowreceiver problem

heskez — Thu, 22 Mar 2018 07:07:37 GMT

NetflowReceiver.0.ip -> Splunk host? OK, thx I'll try and make that correction.

Output netstat -anp | grep 9996 :

tcp 0 0 0.0.0.0:9996 0.0.0.0:* LISTEN 9685/splunkd
udp 0 0 0.0.0.0:9996 0.0.0.0:* 12482/nfcapd

Re: Stream app - Netflowreceiver problem

heskez — Thu, 22 Mar 2018 12:04:16 GMT

I've altered the streamfwd.conf file to and point the netflowreceiver.0.ip to my splunk host where the UDP 9996 is landed. However, even when I try to put in the local ip address or the loopback address, I see this error in streamfwd.log:

2018-03-22 09:20:48 INFO 140441626711808 stream.CaptureServer - Starting data capture
2018-03-22 09:20:48 INFO 140441626711808 stream.SnifferReactor - Starting network capture: sniffer
2018-03-22 09:20:48 ERROR 140441626711808 stream.NetflowReceiver - Caught exception in openDatagramListenersystem:98 bind: Address already in use
2018-03-22 09:20:48 FATAL 140441626711808 stream.CaptureServer - NetflowManager - Unable to start any Netflow Receivers

Re: Stream app - Netflowreceiver problem

gjanders — Thu, 22 Mar 2018 21:14:09 GMT

Ok so try a different port number (just as a test obviously)
netstat -anp | grep 9996 should show you which process is already in use, based on your previous post:
udp 0 0 0.0.0.0:9996 0.0.0.0: 12482/nfcapd

If you ps -ef | grep 12482
You will get some more detail

What is nfcapd? I googled it and it advises it is already capturing network traffic for stream...you will either need to switch port on the stream receiver or stop nfcapd to run the stream on port 9996

Re: Stream app - Netflowreceiver problem

heskez — Fri, 23 Mar 2018 09:02:47 GMT

Thanks, I had another monitoring application configured to collect netflow data indeed 🙂
So I deleted that application and restarted splunkd.
Now I'm getting this in the logs:

domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76

Obviously 192.168.1.2 is the device SENDING netflow data to the splunk host.
It looks like it's sending but the data is bounced because of some misconfiguration somewhere...

Re: Stream app - Netflowreceiver problem

gjanders — Fri, 23 Mar 2018 23:34:40 GMT

I believe I have answered your original question so moving this discussion into an answer.

From what I found, the errors relate to netflow v9 data and I believe it relates to the netflowElement configuration in the streamfwd.conf config file , however in my environment the relevant team switched back to an earlier netflow protocol that did not have these template id data (V7 from memory) and then it just worked...so I never used the netflowElement setup.

Good luck!

Re: Stream app - Netflowreceiver problem

heskez — Tue, 29 Sep 2020 18:38:59 GMT

The thing is now that I've deleted the other collector application, netflow data is coming in :

sourcetype="stream.netflow"

3/24/18
1:04:26.190 PM

{ [-]
app:

bytes: 0

count: 1

dest_ip: 8.8.8.8

dest_port: 53
drop_packet_count: 0

endtime: 2018-03-24T12:04:26.190502Z

packets: 0

packets_in: 1

packets_out: 1

src_ip: 192.168.1.2

src_mac:

src_port: 54808

sum(bytes_in): 73
sum(bytes_out): 73
timestamp: 2018-03-24T12:04:26.190502Z

}
Show as raw text

However, when i click the Stream application nothing is shown in the dashboard, analytics overview or flow visualization. -> No results found.

Re: Stream app - Netflowreceiver problem

heskez — Tue, 29 Sep 2020 18:39:02 GMT

The thing is now that I've deleted the other collector application, netflow data is coming in :

sourcetype="stream.netflow"

3/24/18
1:04:26.190 PM
{ [-]
app:
bytes: 0
count: 1
dest_ip: 8.8.8.8
dest_port: 53
drop_packet_count: 0
endtime: 2018-03-24T12:04:26.190502Z
packets: 0
packets_in: 1
packets_out: 1
src_ip: 192.168.1.2
src_mac:
src_port: 54808
sum(bytes_in): 73
sum(bytes_out): 73
timestamp: 2018-03-24T12:04:26.190502Z
}
Show as raw text

However, when i click the Stream application nothing is shown in the dashboard, analytics overview or flow visualization. -> No results found.