Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

[j_stock]

Hi all,

It doesn't matter how much I read the documentation https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/ConfigureFlowcollector or follow tips from https://answers.splunk.com/answers/636437/how-to-configure-the-splunk-flow-collector-setup-i.htmlhtt... I can't get the TA to ingest netflow from pfSense 2.4.4.

I have pfSense using the sotftflow package exporting netflow ipfix to my combined SH/Indexer (single instance, home setup) on port 9995.

I have the Splunk UF installed on pfSense and it is configured to use a deployment server if needed.
I have SE running on Ubuntu 16.04 v 7.3.1 with Splunk Stream 7.1.3 installed as the app and with the TA.
I have the following configs:

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf:
[streamfwd://streamfwd]
splunk_stream_app_location = https://splunk-enterprise/en-US/app/splunk_app_stream
disabled = false
index = netflow

[streamfwd]
disabled = false
source = stream

[udp://9995]
connection_host = ip
source = stream
index = netflow
disabled = false

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf:
[streamfwd]
port = 8089
ipAddr = 127.0.0.1
netflowReceiver.0.ip = 127.0.0.1
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

UDP 9995 and TCP 8089 are listening and working fine.

I'm hitting walls here. I have no idea what's wrong or whats happening next.

Unusually I get this in streamfwd.log:
2019-08-11 11:16:35 ERROR 140695607523072 stream.CaptureServer - Unable to ping server (19a246f1-d41e-472d-8de4-d42bcfc74f65): /en-US/app/splunk_app_stream/ping/ status=303

I can confirm that /en-US/app/splunk_app_stream/ping/ does not exist... but I have installed from the tgz so I am not sure why it doesn't exist?

Sorry, this is all over the place, as is my config, such is my desperation to get this working.

Please help.

[j_stock · Accepted solution]

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

[NogNeetMachinaa]

Great - will give this a try later this week. I'm struggling with the same thing.

Two questions:

(1) - How would that look like with a forwarder installed on another system then the indexer?

(2) - What would it take to have flow records accepted for both - UDP and TCP?

Cheers - Will

[dconnett_splunk]

This worked for me, thanks @j_stock!

[diogofgm]

check this blog post. It has a nice walkthrough setting up stream in a dist environment.
https://www.splunk.com/blog/2019/02/14/installing-and-managing-splunk-stream-in-a-distributed-enviro...

[j_stock]

Hi,

Thanks for the blog post. I've read that in the past, but it doesn't really address much about netflow.

Also, as the host is pfSense which runs on FreeBSD, the streamfwd binary doesn't run.