I have installed TA-PFSense, sent the logs to the network index with sourcetype pfsense, but none of the fields are being parsed. Do I need to merge the transform.conf or props.conf with the main system or anything else?
This TA has a requirement that you are sending the syslog directly to Splunk. As such, you have to create a UDP listener (Settings > Data Inputs > UDP) on a port (e.g. 5514) and then associate the appropriate sourcetype (pfsense) and index (network) for it to work out-of-box.
I originally tried just sending the syslogs to a file via rsyslog and having Splunk monitor the file. That won't work without modifying the TA.
I have all of these settings configured as you say, but the logs still aren't being parsed.
-Please be sure to have the latest TA-pfsense installed (2.0.5)
-What are the sourcetypes you get?
-The sourcetype pfsense will be rewritten by props.conf/transforms.conf. Check that the TA is on the right Splunk instance that running the parsing phase (refer to this document http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F)
Hi there. I do have version 2.0.5 of TA-pfsense installed.
I'm certain that TA is on the right Splunk instance as I only have one instance of Splunk. This is a brand new Splunk install, and currently I am only sending pfSense logs to it.
I am receiving the log data from pfSense and those events are showing "pfsense:" as their sourcetype. They are being sent to the "network" index. (Is it necessary that they go to the network index?)
I have pushed a new version to splunkbase (2.0.6) , there was a bug in the sourcetyper under default/transforms.conf.
You can use whatever index you want. Just specify one that fits your environment in your inputs.conf.
I updated to 2.0.6 and now my firewall logs are being assigned the sourcetype of "pfsense:filterlog". So that's an improvement.
However, it seems the fields within the logs still aren't being parsed. For example, my latest log line looks like:
Feb 14 08:19:21 filterlog: 5,16777216,,1000000103,bge1,match,block,in,4,0xc0,,46,12426,0,none,1,icmp,184.108.40.206,220.127.116.11,unreachport,18.104.22.168,UDP,5384
Please check that the TA is installed on your search head (if you use distributed search) and that you are not searching in "Fast Mode"
TA is installed on my search head. My environment is not distributed. Just a single Splunk server.
I am searching in "Smart Mode".
I haven't dug into the TA to see how it's built, but I assume that since it takes a given sourcetype (pfsense) and then performs field extractions on it and creates additional sourcetypes (pfsense:logfilter, pfsense:dhcpd, pfsense:webui, etc) that you'd have to modify the TA itself rather significantly to allow it to be used on monitored files.
You could reach out to the TA author and see if s/he responds.
Perhaps someone else can weigh in on how to fix this, I just went ahead and created the UDP listener and it started working great.
(p.s. if my answer was correct for identifying your problem, please mark it as answered)
Oh that's exactly my problem. Do you know what part I will need to modify?
The add-on expects the log data to initially be of sourcetype "pfsense". The add-on will then create new sourcetypes (e.g. "pfsense:filterlog")
Be sure to use version 2.0.2 as there was a bug in version 2.0
I appear to be having an issue where the TA does not appear to be creating proper sourcetypes. I just see 'pfsense:'
try this to extract fields properly
I downvoted this post because this blog post is for the old format of pfsense logs. version 2.2 and above use single-line file formats. this won't work anymore.
I appreciate your answer, but I guess I'm more trying to understand how the app is supposed to work. Should I enter the props.conf/transform.conf entries into splunk manually, or do I have to add what is in the blog on top of the app. If so, what is the point of the app.