Getting Data In

Configuring Splunk_TA_stream 7.1.3 to ingest netflow from pfSense 2.4.4 on SE 7.3.1

j_stock
Explorer

Hi all,

It doesn't matter how much I read the documentation https://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/ConfigureFlowcollector or follow tips from https://answers.splunk.com/answers/636437/how-to-configure-the-splunk-flow-collector-setup-i.htmlhtt... I can't get the TA to ingest netflow from pfSense 2.4.4.

I have pfSense using the sotftflow package exporting netflow ipfix to my combined SH/Indexer (single instance, home setup) on port 9995.

I have the Splunk UF installed on pfSense and it is configured to use a deployment server if needed.
I have SE running on Ubuntu 16.04 v 7.3.1 with Splunk Stream 7.1.3 installed as the app and with the TA.
I have the following configs:

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf:
[streamfwd://streamfwd]
splunk_stream_app_location = https://splunk-enterprise/en-US/app/splunk_app_stream
disabled = false
index = netflow

[streamfwd]
disabled = false
source = stream

[udp://9995]
connection_host = ip
source = stream
index = netflow
disabled = false

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf:
[streamfwd]
port = 8089
ipAddr = 127.0.0.1
netflowReceiver.0.ip = 127.0.0.1
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

UDP 9995 and TCP 8089 are listening and working fine.

I'm hitting walls here. I have no idea what's wrong or whats happening next.

Unusually I get this in streamfwd.log:
2019-08-11 11:16:35 ERROR 140695607523072 stream.CaptureServer - Unable to ping server (19a246f1-d41e-472d-8de4-d42bcfc74f65): /en-US/app/splunk_app_stream/ping/ status=303

I can confirm that /en-US/app/splunk_app_stream/ping/ does not exist... but I have installed from the tgz so I am not sure why it doesn't exist?

Sorry, this is all over the place, as is my config, such is my desperation to get this working.

Please help.

0 Karma
1 Solution

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

View solution in original post

j_stock
Explorer

Right, so I managed to get this working.

/opt/splunk/etc/apps/Splunk_TA_stream/local/streamfwd.conf
[streamfwd]
logConfig = streamfwdlog.conf
port = 8889
ipAddr = 0.0.0.0

netflowReceiver.0.interface = IP
netflowReceiver.0.protocol = udp
netflowReceiver.0.port = 9995
netflowReceiver.0.decoder = netflow

/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:443/en-us/custom/splunk_app_stream/
stream_forwarder_id =
disabled = 0

[udp://9995]
connection_host = ip
disabled = 0
index = stream
source = stream

This is now allowing netflow to both be accepted and indexed correctly by Splunk_TA_stream with the flow being delivered by softflowd within pfSense.

Hopefully this helps someone else down the line.

Cheers

NogNeetMachinaa
Explorer

Great - will give this a try later this week. I'm struggling with the same thing.

 

Two questions:

(1) - How would that look like with a forwarder installed on another system then the indexer?

(2) - What would it take to have flow records accepted for both - UDP and TCP?

 

Cheers - Will

0 Karma

dconnett_splunk
Splunk Employee
Splunk Employee

This worked for me, thanks @j_stock!

0 Karma

diogofgm
SplunkTrust
SplunkTrust

check this blog post. It has a nice walkthrough setting up stream in a dist environment.
https://www.splunk.com/blog/2019/02/14/installing-and-managing-splunk-stream-in-a-distributed-enviro...

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

j_stock
Explorer

Hi,

Thanks for the blog post. I've read that in the past, but it doesn't really address much about netflow.

Also, as the host is pfSense which runs on FreeBSD, the streamfwd binary doesn't run.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...