All Apps and Add-ons

Why is TA-pfsense 2.5 not parsing events?

TroyF
New Member

I've recently installed Splunk to begin learning how to use it, and the first thing I wanted to do was parse the logs from a pfsense firewall.  I believe that the TA-pfsense application is meant to help parse the syslog information, but despite my best efforts I cannot get it working.

My environment has the following:

Splunk - 8.1.3 (single instance)

pfSense - 21.02.2 sending logs in syslog format

TA-pfsense v2.5 release March 3, 2021

Splunk is receiving the syslog events into an index called 'network' and the events are labelled with the default pfsense sourcetype but this is not being parsed into the various other types of pfsense:filterlog, pfsense:unbound etc.

I grabbed the REGEX string from transforms.conf and did some testing against the events getting pulled into Splunk, it seems like the string is not formatted for the logs I have.

I made the following changes:

Original: REGEX = \w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2}\s(?:[\w.]+\s)?(\w+)

Updated: REGEX = \w{1,3}\s\w{4}-\w{1,2}-\w{1,2}T\d{1,2}:\d{1,2}:\d{1,2}.\d{1,6}-\d{1,2}:\w{1,2}\s\w+.?\w+.?\w+(?:[\w.]+\s)?(\w+)

Admittedly I am very new to regex and so the above might be less than ideal, but it does seem to parse out the sourcetype.  However after crossing that hurdle it seems like all of the EXTRACT statements also don't match the log format Splunk is gathering.

Is anyone else running a current version of pfSense with the latest TA-pfsense application and having similar issues? 

Any pointers would be appreciated.  I've searched around but have not seen any current posts with people reporting a similar issue.

Thanks!

 

 

Labels (1)
0 Karma

ozsimon
Observer

This still appears to be an issue, fresh install today :

Splunk Enterprise Version: 9.0.2
Pfsense CE 2.6.0-RELEASE (amd64)
TA-Pfsense 2.5.1
sourcetype = pfsense

and not processing events.

I've tried setting the REGEX = \w{1,3}\s\w{4}-\w{1,2}-\w{1,2}T\d{1,2}:\d{1,2}:\d{1,2}.\d{1,6}-\d{1,2}:\w{1,2}\s\w+.?\w+.?\w+(?:[\w.]+\s)?(\w+) as per one 'solved' answer but then found that SEDCMD-event_cleaner doesn't appear to be removing the date but I am getting it slightly transformed as sorcetype = pfsense.Nov (for November I presume)

Will continue to explore this week as time allows.

---------------------------------------

As a follow up I then messed with TA-Pfsense-main, remove both, reinstalled TA-pfsense and it just lept into life - I can't for the life of me work out why, nothing else was changed!!

0 Karma

ReignInChaos
Loves-to-Learn Lots

Adding into the thread as I am too having issues similar as reported.  I was at one point processing logs and extracting fields as expected, however, stopped a few days ago.  In outside tutorials and posts, I noted the timestamp listed in the logs is much different thank what I am seeing.

Some sites show timestamp as:

MMM d HH:MM:SS filterlog: {etc}

However, my installation is producing logs in the following format:

1 YYYY-MM-DDTHH:MM:SS.084257+00:00 {pfSense DNS name} filterlog {etc}

Following recommendations from other posts, it seemed the transforms.conf regular expression did not take into account this detailed timestamp.  I updated as follows and I am now extracting accurate source types but still working to get fields properly extracted through editing props.conf.

\w{1}\s\w{4}-\w{1,2}-\w{1,2}T\d{1,2}:\d{1,2}:\d{1,2}\.\d{1,6}\+\d{1,2}:\d{1,2}\s(?:[\w.]+\s)?(\w+)

I will post a follow up once I make more progress.

0 Karma

rg33
Explorer

For me, it was because I had  pfsense configured for syslog style logs.

After I changed status/system logs/settings back to: 

  • Log Message Format: BSD (RFC 3164, default)

Splunk started to register the different source types as expected.

Kind regards,
rg

0 Karma

wjclinton
New Member

i am having similar issues with pfsense 21.05 and ta-pfsense 2.50.  it is not extracting the various fields like bytes_*. 

it works correctly with pfsense 2.45 

0 Karma

samsclub91
New Member

Bump!  I'm having similar issues.  Running pfSense CE 2.5.1 with TA-pfsense 2.5.0

 

I too have searched around and found various different things to try none of which have worked.  I'm going to try sending a message to the TA-pfsense dev too but thought I would bump this post too.

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...