Getting Data In

Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock
Explorer

Hi all,

It doesn't matter how much I read the documentation https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/ConfigureFlowcollector or follow tips from https://answers.splunk.com/answers/636437/how-to-configure-the-splunk-flow-collector-setup-i.htmlhtt... I can't get the TA to ingest netflow from pfSense 2.4.4.

I have pfSense using the sotftflow package exporting netflow ipfix to my combined SH/Indexer (single instance, home setup) on port 9995.

I have the Splunk UF installed on pfSense and it is configured to use a deployment server if needed.
I have SE running on Ubuntu 16.04 v 7.3.1 with Splunk Stream 7.1.3 installed as the app and with the TA.
I have the following configs:

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf:
[streamfwd://streamfwd]
splunk_stream_app_location = https://splunk-enterprise/en-US/app/splunk_app_stream
disabled = false
index = netflow

[streamfwd]
disabled = false
source = stream

[udp://9995]
connection_host = ip
source = stream
index = netflow
disabled = false

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf:
[streamfwd]
port = 8089
ipAddr = 127.0.0.1
netflowReceiver.0.ip = 127.0.0.1
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

UDP 9995 and TCP 8089 are listening and working fine.

I'm hitting walls here. I have no idea what's wrong or whats happening next.

Unusually I get this in streamfwd.log:
2019-08-11 11:16:35 ERROR 140695607523072 stream.CaptureServer - Unable to ping server (19a246f1-d41e-472d-8de4-d42bcfc74f65): /en-US/app/splunk_app_stream/ping/ status=303

I can confirm that /en-US/app/splunk_app_stream/ping/ does not exist... but I have installed from the tgz so I am not sure why it doesn't exist?

Sorry, this is all over the place, as is my config, such is my desperation to get this working.

Please help.

0 Karma
1 Solution

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

View solution in original post

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

NogNeetMachinaa
Explorer

Great - will give this a try later this week. I'm struggling with the same thing.

 

Two questions:

(1) - How would that look like with a forwarder installed on another system then the indexer?

(2) - What would it take to have flow records accepted for both - UDP and TCP?

 

Cheers - Will

0 Karma

dconnett_splunk
Splunk Employee
Splunk Employee

This worked for me, thanks @j_stock!

0 Karma

diogofgm
SplunkTrust
SplunkTrust

check this blog post. It has a nice walkthrough setting up stream in a dist environment.
https://www.splunk.com/blog/2019/02/14/installing-and-managing-splunk-stream-in-a-distributed-enviro...

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

j_stock
Explorer

Hi,

Thanks for the blog post. I've read that in the past, but it doesn't really address much about netflow.

Also, as the host is pfSense which runs on FreeBSD, the streamfwd binary doesn't run.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...