topic Re: TA-pfsense: Why are none of the fields being parsed? in All Apps and Add-ons

TA-pfsense: Why are none of the fields being parsed?

Epicism1 — Wed, 01 Apr 2015 02:10:40 GMT

Hello,

I have installed TA-PFSense, sent the logs to the network index with sourcetype pfsense, but none of the fields are being parsed. Do I need to merge the transform.conf or props.conf with the main system or anything else?

Thank you.

Re: TA-pfsense: Why are none of the fields being parsed?

kml_uvce — Wed, 01 Apr 2015 03:48:03 GMT

try this to extract fields properly

http://blog.basementpctech.com/2012/02/splunk-and-pfsense-what-pair.html

Re: TA-pfsense: Why are none of the fields being parsed?

Epicism1 — Wed, 01 Apr 2015 11:15:47 GMT

I appreciate your answer, but I guess I'm more trying to understand how the app is supposed to work. Should I enter the props.conf/transform.conf entries into splunk manually, or do I have to add what is in the blog on top of the app. If so, what is the point of the app.

Re: TA-pfsense: Why are none of the fields being parsed?

my2ndhead — Mon, 06 Apr 2015 10:11:17 GMT

The add-on expects the log data to initially be of sourcetype "pfsense". The add-on will then create new sourcetypes (e.g. "pfsense:filterlog")

Be sure to use version 2.0.2 as there was a bug in version 2.0

Re: TA-pfsense: Why are none of the fields being parsed?

pickerin — Tue, 20 Oct 2015 10:21:56 GMT

This TA has a requirement that you are sending the syslog directly to Splunk. As such, you have to create a UDP listener (Settings > Data Inputs > UDP) on a port (e.g. 5514) and then associate the appropriate sourcetype (pfsense) and index (network) for it to work out-of-box.

I originally tried just sending the syslogs to a file via rsyslog and having Splunk monitor the file. That won't work without modifying the TA.

Re: TA-pfsense: Why are none of the fields being parsed?

Epicism1 — Tue, 20 Oct 2015 12:59:39 GMT

Oh that's exactly my problem. Do you know what part I will need to modify?

Re: TA-pfsense: Why are none of the fields being parsed?

pickerin — Tue, 20 Oct 2015 13:03:00 GMT

I haven't dug into the TA to see how it's built, but I assume that since it takes a given sourcetype (pfsense) and then performs field extractions on it and creates additional sourcetypes (pfsense:logfilter, pfsense:dhcpd, pfsense:webui, etc) that you'd have to modify the TA itself rather significantly to allow it to be used on monitored files.

You could reach out to the TA author and see if s/he responds.

Perhaps someone else can weigh in on how to fix this, I just went ahead and created the UDP listener and it started working great.

(p.s. if my answer was correct for identifying your problem, please mark it as answered)

Re: TA-pfsense: Why are none of the fields being parsed?

nickatripp — Tue, 09 Feb 2016 20:10:42 GMT

I downvoted this post because this blog post is for the old format of pfsense logs. version 2.2 and above use single-line file formats. this won't work anymore.

Re: TA-pfsense: Why are none of the fields being parsed?

nickatripp — Tue, 09 Feb 2016 20:11:40 GMT

I have all of these settings configured as you say, but the logs still aren't being parsed.

Re: TA-pfsense: Why are none of the fields being parsed?

my2ndhead — Tue, 09 Feb 2016 21:27:14 GMT

-Please be sure to have the latest TA-pfsense installed (2.0.5)
-What are the sourcetypes you get?
-The sourcetype pfsense will be rewritten by props.conf/transforms.conf. Check that the TA is on the right Splunk instance that running the parsing phase (refer to this document http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F)

Re: TA-pfsense: Why are none of the fields being parsed?

nickatripp — Tue, 09 Feb 2016 23:29:26 GMT

Hi there. I do have version 2.0.5 of TA-pfsense installed.

I'm certain that TA is on the right Splunk instance as I only have one instance of Splunk. This is a brand new Splunk install, and currently I am only sending pfSense logs to it.

I am receiving the log data from pfSense and those events are showing "pfsense:" as their sourcetype. They are being sent to the "network" index. (Is it necessary that they go to the network index?)

Re: TA-pfsense: Why are none of the fields being parsed?

xECK29x — Sat, 13 Feb 2016 03:01:39 GMT

I appear to be having an issue where the TA does not appear to be creating proper sourcetypes. I just see 'pfsense:'

Re: TA-pfsense: Why are none of the fields being parsed?

my2ndhead — Sun, 14 Feb 2016 09:43:05 GMT

I have pushed a new version to splunkbase (2.0.6) , there was a bug in the sourcetyper under default/transforms.conf.

You can use whatever index you want. Just specify one that fits your environment in your inputs.conf.

Re: TA-pfsense: Why are none of the fields being parsed?

nickatripp — Sun, 14 Feb 2016 13:23:10 GMT

Thanks!

I updated to 2.0.6 and now my firewall logs are being assigned the sourcetype of "pfsense:filterlog". So that's an improvement.

However, it seems the fields within the logs still aren't being parsed. For example, my latest log line looks like:

Feb 14 08:19:21 filterlog: 5,16777216,,1000000103,bge1,match,block,in,4,0xc0,,46,12426,0,none,1,icmp,1.1.1.1,2.2.2.2,unreachport,1.1.1.1,UDP,5384

Re: TA-pfsense: Why are none of the fields being parsed?

my2ndhead — Sun, 14 Feb 2016 15:20:45 GMT

Please check that the TA is installed on your search head (if you use distributed search) and that you are not searching in "Fast Mode"

Re: TA-pfsense: Why are none of the fields being parsed?

nickatripp — Sun, 14 Feb 2016 19:18:35 GMT

TA is installed on my search head. My environment is not distributed. Just a single Splunk server.

I am searching in "Smart Mode".