Stream app - Netflowreceiver problem

[heskez]

Hi There, I've configured the stream app and streamfwd.log as follow:

netflowReceiver.0.ip = 192.168.1.2
netflowReceiver.0.port = 9996
netflowReceiver.0.protocol = udp
netflowReceiver.0.decoder = netflow

UDP Netflow is coming in on the splunk server, confirmed with TCPDUMP

However, I don't get the netflow data and see these kinds of errors in streamfwd.log:

Caught exception in openDatagramListenersystem:99 bind
Unable to start any Netflow Receivers

[gjanders · Accepted solution]

Ok so try a different port number (just as a test obviously)
netstat -anp | grep 9996 should show you which process is already in use, based on your previous post:
udp 0 0 0.0.0.0:9996 0.0.0.0: 12482/nfcapd

If you ps -ef | grep 12482
You will get some more detail

What is nfcapd? I googled it and it advises it is already capturing network traffic for stream...you will either need to switch port on the stream receiver or stop nfcapd to run the stream on port 9996

[gjanders]

Would you mind accepting this answer now that the question is resolved?

Thanks

[heskez]

Yes, this topic is answered.

[gjanders]

Actually there is a button on the answer (near add comment) that says "Accept answer" this marks the question as answered so those browsing the forum don't attempt to re-answer.

It also awards karma points 🙂

[heskez]

Thanks, I had another monitoring application configured to collect netflow data indeed 🙂
So I deleted that application and restarted splunkd.
Now I'm getting this in the logs:

domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76
2018-03-23 09:54:56 WARN 140372033984256 stream.NetflowReceiver - NetFlowDecoder::decodeFlow Unable to decode flow set data. No template with id 262 received for observation domain id 1 from device 192.168.1.2 . Dropping flow data set of size 76

Obviously 192.168.1.2 is the device SENDING netflow data to the splunk host.
It looks like it's sending but the data is bounced because of some misconfiguration somewhere...

[gjanders]

I believe I have answered your original question so moving this discussion into an answer.

From what I found, the errors relate to netflow v9 data and I believe it relates to the netflowElement configuration in the streamfwd.conf config file , however in my environment the relevant team switched back to an earlier netflow protocol that did not have these template id data (V7 from memory) and then it just worked...so I never used the netflowElement setup.

Good luck!

[heskez]

The thing is now that I've deleted the other collector application, netflow data is coming in :

sourcetype="stream.netflow"

3/24/18
1:04:26.190 PM
{ [-]
app:
bytes: 0
count: 1
dest_ip: 8.8.8.8
dest_port: 53
drop_packet_count: 0
endtime: 2018-03-24T12:04:26.190502Z
packets: 0
packets_in: 1
packets_out: 1
src_ip: 192.168.1.2
src_mac:
src_port: 54808
sum(bytes_in): 73
sum(bytes_out): 73
timestamp: 2018-03-24T12:04:26.190502Z
}
Show as raw text

However, when i click the Stream application nothing is shown in the dashboard, analytics overview or flow visualization. -> No results found.

[gjanders]

I think we've gone far enough in this one question, perhaps that could be a new question or perhaps Splunk support might help here.

I'd suspect the Splunk server your using doesn't have access to the indexes containing the relevant data however at this point I am guessing!

[gjanders]

Is this an independent stream forwarder and are you running the process as root? Or as a non-root user?
Also you have a Splunk instance with the webgui enabled and the stream application on the same host?

[heskez]

Yes, Everything, the Splunk instance with the web gui, the stream application is installed on one host.
The process runs as root.

[gjanders]

Ok so the IP 192.168.1.2 is configured on the server?
Does netstat -an | grep 9996 show anything listening on that port?

I do not see netflowReceiver.0.protocol = udp as a mentioned line in Configure netflow collector but not sure if that is making any difference here (netflow is normally udp so you should be able to drop this line).

Finally when you said:
"Hi There, I've configured the stream app and streamfwd.log as follow: ", I assume you mean streamfwd.conf ?

[heskez]

The 192.168.1.2 is the ip address of the firewall which is sending the netflow information to Splunk machine.

Output on splunk: udp 0 0 0.0.0.0:9996 0.0.0.0:*

Yes offcourse streamfwd.conf 🙂

[gjanders]

netflowReceiver.0.ip = should be the IP of the host listening for the data (i.e. your server), not the IP of the server sending the data...

[heskez]

NetflowReceiver.0.ip -> Splunk host? OK, thx I'll try and make that correction.

Output netstat -anp | grep 9996 :

tcp 0 0 0.0.0.0:9996 0.0.0.0:* LISTEN 9685/splunkd
udp 0 0 0.0.0.0:9996 0.0.0.0:* 12482/nfcapd

[heskez]

I've altered the streamfwd.conf file to and point the netflowreceiver.0.ip to my splunk host where the UDP 9996 is landed. However, even when I try to put in the local ip address or the loopback address, I see this error in streamfwd.log:

2018-03-22 09:20:48 INFO 140441626711808 stream.CaptureServer - Starting data capture
2018-03-22 09:20:48 INFO 140441626711808 stream.SnifferReactor - Starting network capture: sniffer
2018-03-22 09:20:48 ERROR 140441626711808 stream.NetflowReceiver - Caught exception in openDatagramListenersystem:98 bind: Address already in use
2018-03-22 09:20:48 FATAL 140441626711808 stream.CaptureServer - NetflowManager - Unable to start any Netflow Receivers

[heskez]

The thing is now that I've deleted the other collector application, netflow data is coming in :

sourcetype="stream.netflow"

3/24/18
1:04:26.190 PM

{ [-]
app:

bytes: 0

count: 1

dest_ip: 8.8.8.8

dest_port: 53
drop_packet_count: 0

endtime: 2018-03-24T12:04:26.190502Z

packets: 0

packets_in: 1

packets_out: 1

src_ip: 192.168.1.2

src_mac:

src_port: 54808

sum(bytes_in): 73
sum(bytes_out): 73
timestamp: 2018-03-24T12:04:26.190502Z

}
Show as raw text

However, when i click the Stream application nothing is shown in the dashboard, analytics overview or flow visualization. -> No results found.

[gjanders]

Also can you run a netstat -anp | grep 9996
That should show you what process is using 9996

[heskez]

Was this inputs.conf sufficient in order to proceed?

[heskez]

Anyone out there?

[heskez]

Hello there?