All Apps and Add-ons

TA-pfsense: Why are none of the fields being parsed?

Epicism1
Explorer

Hello,

I have installed TA-PFSense, sent the logs to the network index with sourcetype pfsense, but none of the fields are being parsed. Do I need to merge the transform.conf or props.conf with the main system or anything else?

Thank you.

pickerin
Path Finder

This TA has a requirement that you are sending the syslog directly to Splunk. As such, you have to create a UDP listener (Settings > Data Inputs > UDP) on a port (e.g. 5514) and then associate the appropriate sourcetype (pfsense) and index (network) for it to work out-of-box.

I originally tried just sending the syslogs to a file via rsyslog and having Splunk monitor the file. That won't work without modifying the TA.

nickatripp
Explorer

I have all of these settings configured as you say, but the logs still aren't being parsed.

0 Karma

my2ndhead
SplunkTrust
SplunkTrust

-Please be sure to have the latest TA-pfsense installed (2.0.5)
-What are the sourcetypes you get?
-The sourcetype pfsense will be rewritten by props.conf/transforms.conf. Check that the TA is on the right Splunk instance that running the parsing phase (refer to this document http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F)

0 Karma

nickatripp
Explorer

Hi there. I do have version 2.0.5 of TA-pfsense installed.

I'm certain that TA is on the right Splunk instance as I only have one instance of Splunk. This is a brand new Splunk install, and currently I am only sending pfSense logs to it.

I am receiving the log data from pfSense and those events are showing "pfsense:" as their sourcetype. They are being sent to the "network" index. (Is it necessary that they go to the network index?)

0 Karma

my2ndhead
SplunkTrust
SplunkTrust

I have pushed a new version to splunkbase (2.0.6) , there was a bug in the sourcetyper under default/transforms.conf.

You can use whatever index you want. Just specify one that fits your environment in your inputs.conf.

0 Karma

nickatripp
Explorer

Thanks!

I updated to 2.0.6 and now my firewall logs are being assigned the sourcetype of "pfsense:filterlog". So that's an improvement.

However, it seems the fields within the logs still aren't being parsed. For example, my latest log line looks like:

Feb 14 08:19:21 filterlog: 5,16777216,,1000000103,bge1,match,block,in,4,0xc0,,46,12426,0,none,1,icmp,1.1.1.1,2.2.2.2,unreachport,1.1.1.1,UDP,5384
0 Karma

my2ndhead
SplunkTrust
SplunkTrust

Please check that the TA is installed on your search head (if you use distributed search) and that you are not searching in "Fast Mode"

0 Karma

nickatripp
Explorer

TA is installed on my search head. My environment is not distributed. Just a single Splunk server.

I am searching in "Smart Mode".

0 Karma

pickerin
Path Finder

I haven't dug into the TA to see how it's built, but I assume that since it takes a given sourcetype (pfsense) and then performs field extractions on it and creates additional sourcetypes (pfsense:logfilter, pfsense:dhcpd, pfsense:webui, etc) that you'd have to modify the TA itself rather significantly to allow it to be used on monitored files.

You could reach out to the TA author and see if s/he responds.

Perhaps someone else can weigh in on how to fix this, I just went ahead and created the UDP listener and it started working great.

(p.s. if my answer was correct for identifying your problem, please mark it as answered)

0 Karma

Epicism1
Explorer

Oh that's exactly my problem. Do you know what part I will need to modify?

0 Karma

my2ndhead
SplunkTrust
SplunkTrust

The add-on expects the log data to initially be of sourcetype "pfsense". The add-on will then create new sourcetypes (e.g. "pfsense:filterlog")

Be sure to use version 2.0.2 as there was a bug in version 2.0

xECK29x
Engager

I appear to be having an issue where the TA does not appear to be creating proper sourcetypes. I just see 'pfsense:'

0 Karma

kml_uvce
Builder
0 Karma

nickatripp
Explorer

I downvoted this post because this blog post is for the old format of pfsense logs. version 2.2 and above use single-line file formats. this won't work anymore.

0 Karma

Epicism1
Explorer

I appreciate your answer, but I guess I'm more trying to understand how the app is supposed to work. Should I enter the props.conf/transform.conf entries into splunk manually, or do I have to add what is in the blog on top of the app. If so, what is the point of the app.

Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!