topic Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1 in Getting Data In

Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock — Wed, 30 Sep 2020 01:42:42 GMT

Hi all,

It doesn't matter how much I read the documentation https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/ConfigureFlowcollector or follow tips from https://answers.splunk.com/answers/636437/how-to-configure-the-splunk-flow-collector-setup-i.htmlhttps://answers.splunk.com/answers/743408/streamfwd-is-not-forwarding-netflow-v9-data-to-sh.html I can't get the TA to ingest netflow from pfSense 2.4.4.

I have pfSense using the sotftflow package exporting netflow ipfix to my combined SH/Indexer (single instance, home setup) on port 9995.

I have the Splunk UF installed on pfSense and it is configured to use a deployment server if needed.
I have SE running on Ubuntu 16.04 v 7.3.1 with Splunk Stream 7.1.3 installed as the app and with the TA.
I have the following configs:

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf:
[streamfwd://streamfwd]
splunk_stream_app_location = https://splunk-enterprise/en-US/app/splunk_app_stream
disabled = false
index = netflow

[streamfwd]
disabled = false
source = stream

[udp://9995]
connection_host = ip
source = stream
index = netflow
disabled = false

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf:
[streamfwd]
port = 8089
ipAddr = 127.0.0.1
netflowReceiver.0.ip = 127.0.0.1
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

UDP 9995 and TCP 8089 are listening and working fine.

I'm hitting walls here. I have no idea what's wrong or whats happening next.

Unusually I get this in streamfwd.log:
2019-08-11 11:16:35 ERROR 140695607523072 stream.CaptureServer - Unable to ping server (19a246f1-d41e-472d-8de4-d42bcfc74f65): /en-US/app/splunk_app_stream/ping/ status=303

I can confirm that /en-US/app/splunk_app_stream/ping/ does not exist... but I have installed from the tgz so I am not sure why it doesn't exist?

Sorry, this is all over the place, as is my config, such is my desperation to get this working.

Please help.

Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

diogofgm — Mon, 12 Aug 2019 01:23:31 GMT

check this blog post. It has a nice walkthrough setting up stream in a dist environment.
https://www.splunk.com/blog/2019/02/14/installing-and-managing-splunk-stream-in-a-distributed-environment.html

Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock — Mon, 12 Aug 2019 09:42:41 GMT

Hi,

Thanks for the blog post. I've read that in the past, but it doesn't really address much about netflow.

Also, as the host is pfSense which runs on FreeBSD, the streamfwd binary doesn't run.

Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock — Wed, 30 Sep 2020 01:44:35 GMT

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

dconnett_splunk — Mon, 26 Oct 2020 21:25:07 GMT

This worked for me, thanks @j_stock!

Re: Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

NogNeetMachinaa — Sun, 09 May 2021 07:24:17 GMT

Great - will give this a try later this week. I'm struggling with the same thing.

Two questions:

(1) - How would that look like with a forwarder installed on another system then the indexer?

(2) - What would it take to have flow records accepted for both - UDP and TCP?

Cheers - Will