TA-pfsense: Why are none of the fields being parsed?

[Epicism1]

Hello,

I have installed TA-PFSense, sent the logs to the network index with sourcetype pfsense, but none of the fields are being parsed. Do I need to merge the transform.conf or props.conf with the main system or anything else?

Thank you.

[pickerin]

This TA has a requirement that you are sending the syslog directly to Splunk. As such, you have to create a UDP listener (Settings > Data Inputs > UDP) on a port (e.g. 5514) and then associate the appropriate sourcetype (pfsense) and index (network) for it to work out-of-box.

I originally tried just sending the syslogs to a file via rsyslog and having Splunk monitor the file. That won't work without modifying the TA.

[nickatripp]

I have all of these settings configured as you say, but the logs still aren't being parsed.

[my2ndhead]

-Please be sure to have the latest TA-pfsense installed (2.0.5)
-What are the sourcetypes you get?
-The sourcetype pfsense will be rewritten by props.conf/transforms.conf. Check that the TA is on the right Splunk instance that running the parsing phase (refer to this document http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F)

[nickatripp]

Hi there. I do have version 2.0.5 of TA-pfsense installed.

I'm certain that TA is on the right Splunk instance as I only have one instance of Splunk. This is a brand new Splunk install, and currently I am only sending pfSense logs to it.

I am receiving the log data from pfSense and those events are showing "pfsense:" as their sourcetype. They are being sent to the "network" index. (Is it necessary that they go to the network index?)

[my2ndhead]

I have pushed a new version to splunkbase (2.0.6) , there was a bug in the sourcetyper under default/transforms.conf.

You can use whatever index you want. Just specify one that fits your environment in your inputs.conf.

[nickatripp]

Thanks!

I updated to 2.0.6 and now my firewall logs are being assigned the sourcetype of "pfsense:filterlog". So that's an improvement.

However, it seems the fields within the logs still aren't being parsed. For example, my latest log line looks like:

Feb 14 08:19:21 filterlog: 5,16777216,,1000000103,bge1,match,block,in,4,0xc0,,46,12426,0,none,1,icmp,1.1.1.1,2.2.2.2,unreachport,1.1.1.1,UDP,5384

[my2ndhead]

Please check that the TA is installed on your search head (if you use distributed search) and that you are not searching in "Fast Mode"

[nickatripp]

TA is installed on my search head. My environment is not distributed. Just a single Splunk server.

I am searching in "Smart Mode".

[pickerin]

I haven't dug into the TA to see how it's built, but I assume that since it takes a given sourcetype (pfsense) and then performs field extractions on it and creates additional sourcetypes (pfsense:logfilter, pfsense:dhcpd, pfsense:webui, etc) that you'd have to modify the TA itself rather significantly to allow it to be used on monitored files.

You could reach out to the TA author and see if s/he responds.

Perhaps someone else can weigh in on how to fix this, I just went ahead and created the UDP listener and it started working great.

(p.s. if my answer was correct for identifying your problem, please mark it as answered)

[Epicism1]

Oh that's exactly my problem. Do you know what part I will need to modify?

[my2ndhead]

The add-on expects the log data to initially be of sourcetype "pfsense". The add-on will then create new sourcetypes (e.g. "pfsense:filterlog")

Be sure to use version 2.0.2 as there was a bug in version 2.0

[xECK29x]

I appear to be having an issue where the TA does not appear to be creating proper sourcetypes. I just see 'pfsense:'

[kml_uvce]

try this to extract fields properly

http://blog.basementpctech.com/2012/02/splunk-and-pfsense-what-pair.html

[nickatripp]

I downvoted this post because this blog post is for the old format of pfsense logs. version 2.2 and above use single-line file formats. this won't work anymore.

[Epicism1]

I appreciate your answer, but I guess I'm more trying to understand how the app is supposed to work. Should I enter the props.conf/transform.conf entries into splunk manually, or do I have to add what is in the blog on top of the app. If so, what is the point of the app.