Getting Data In

Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock
Explorer

Hi all,

It doesn't matter how much I read the documentation https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/ConfigureFlowcollector or follow tips from https://answers.splunk.com/answers/636437/how-to-configure-the-splunk-flow-collector-setup-i.htmlhtt... I can't get the TA to ingest netflow from pfSense 2.4.4.

I have pfSense using the sotftflow package exporting netflow ipfix to my combined SH/Indexer (single instance, home setup) on port 9995.

I have the Splunk UF installed on pfSense and it is configured to use a deployment server if needed.
I have SE running on Ubuntu 16.04 v 7.3.1 with Splunk Stream 7.1.3 installed as the app and with the TA.
I have the following configs:

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf:
[streamfwd://streamfwd]
splunk_stream_app_location = https://splunk-enterprise/en-US/app/splunk_app_stream
disabled = false
index = netflow

[streamfwd]
disabled = false
source = stream

[udp://9995]
connection_host = ip
source = stream
index = netflow
disabled = false

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf:
[streamfwd]
port = 8089
ipAddr = 127.0.0.1
netflowReceiver.0.ip = 127.0.0.1
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

UDP 9995 and TCP 8089 are listening and working fine.

I'm hitting walls here. I have no idea what's wrong or whats happening next.

Unusually I get this in streamfwd.log:
2019-08-11 11:16:35 ERROR 140695607523072 stream.CaptureServer - Unable to ping server (19a246f1-d41e-472d-8de4-d42bcfc74f65): /en-US/app/splunk_app_stream/ping/ status=303

I can confirm that /en-US/app/splunk_app_stream/ping/ does not exist... but I have installed from the tgz so I am not sure why it doesn't exist?

Sorry, this is all over the place, as is my config, such is my desperation to get this working.

Please help.

0 Karma
1 Solution

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

View solution in original post

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

NogNeetMachinaa
Explorer

Great - will give this a try later this week. I'm struggling with the same thing.

 

Two questions:

(1) - How would that look like with a forwarder installed on another system then the indexer?

(2) - What would it take to have flow records accepted for both - UDP and TCP?

 

Cheers - Will

0 Karma

dconnett_splunk
Splunk Employee
Splunk Employee

This worked for me, thanks @j_stock!

0 Karma

diogofgm
SplunkTrust
SplunkTrust

check this blog post. It has a nice walkthrough setting up stream in a dist environment.
https://www.splunk.com/blog/2019/02/14/installing-and-managing-splunk-stream-in-a-distributed-enviro...

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

j_stock
Explorer

Hi,

Thanks for the blog post. I've read that in the past, but it doesn't really address much about netflow.

Also, as the host is pfSense which runs on FreeBSD, the streamfwd binary doesn't run.

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...