Parsing pfSense Logs Part 2

[pstutz]

Someone else asked this question already but no answers were posted... I am running pfSense 2.0-RC2 which produces mulitline firewall logs. Shown below are six lines which should be merged into three. To test my props.conf setup I used the word "match" as the pattern for "BREAK_ONLY_BEFORE". This did not work and so after searching the forums and hours of tweaking I need some guidance. My props.conf looks like this:

[pfsense_pf]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = match

also tried...

[pfsense_pf]

SHOULD_LINEMERGE = true

LINE_BREAKER= match

Log sample:

May 29 15:10:21 pfsense pf: 00:00:02.724781 rule 64/0(match): pass in on vr2: (tos 0x0, ttl 128, id 20219, offset 0, flags [DF], proto TCP (6), length 48)

May 29 15:10:21 pfsense pf:     10.0.1.251.59417 > 192.168.1.3.9997: Flags [S], cksum 0xc2b5 (correct), seq 1434545691, win 8192, options [mss 1460,nop,nop,sackOK], length 0

May 29 15:10:24 pfsense pf: 00:00:02.962391 rule 45/0(match): block in on vr1: (tos 0x20, ttl 114, id 14617, offset 0, flags [none], proto UDP (17), length 59)

May 29 15:10:24 pfsense pf:     92.14.58.97.34192 > 67.183.148.198.12996: UDP, length 31

May 29 15:10:30 pfsense pf: 00:00:05.301471 rule 45/0(match): block in on vr1: (tos 0x0, ttl 255, id 31114, offset 0, flags [none], proto UDP (17), length 331)

May 29 15:10:30 pfsense pf:     73.98.34.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 303, xid 0xeb5408e8, Flags [Broadcast]

Testing Steps:

1. Tweak props.conf, restart splunk.
2. Define log file to process/monitor, set sourcetype manually to pfsense_pf.
3. Browse results in search app.
4. If successful stop, else delete input source go to step 1.

[dygobel]

The solution is and I have this working in pfsense 2.1

Go to Diagnostics -> Edit File, open file /etc/inc/filter.inc and change the following:

From: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | logger -t pf -p local0.info");
To: mwexec_bg("/usr/sbin/tcpdump -s 256 -v -l -n -e -ttt -i pflog0 | /usr/bin/sed -l -e 'N;s/\n //;P;D;' | logger -t pf -p local0.info");

You have to reboot your pfsense to see the changes applied, and there no more split lines!

My question now is, howto normalize it in Splunk?

NOTE: credits goes to: http://thwack.solarwinds.com/thread/54381

[dre]

The problem is that pfSense is calling tcpdump with the -v option, which now includes a newline in the output, when formerly it did not (!!). You can probably see the same results by running tcpdump on your local machine with -v.

More info about this seemingly unwanted line break: http://seclists.org/tcpdump/2010/q1/16

There was some discussion of reverting this change, but that was two years ago 😕 The info provided by -v is nice, just not the line break... some pfSense folk are now simply filtering the line break before handing off to syslog, e.g. with sed.

Details on filtering the line break in pfSense: http://redmine.pfsense.org/issues/1938

Personally I think I'd rather patch tcpdump to get rid of the line break that wasn't there to begin with for -v mode, as Apple did. To see this simple change, search the following page for the string: gflag

http://opensource.apple.com/source/tcpdump/tcpdump-32/tcpdump/print-ip.c

Cheers,
-dre

[s0mmy1]

I just created a blog entry on how I was able to parse the pfSense files. It works for me and hopefully will work for you too. http://blog.basementpctech.com/2012/02/splunk-and-pfsense-what-pair.html

I made changes to the props.conf and the transforms.conf files

[JasonStewarSplu]

It is a one-line change to make pfSense put it in a single syslog transaction (like everything else does), and that fixes it not just for Splunk, but for every syslog receiver.

[JasonStewarSplu]

This is not a problem that can be properly fixed on the Splunk side. Lguinn's solution would be acceptable if you were only logging firewall events, but if you logged other pfSense events, they would be trailed onto the previous firewall event until the next "match".

The fix would be to get the pfSense guys to take the line break out of their tcpdump command. It is causing problems with every syslog receiver out there.

They are already discussing it, and there is a workaround if you are running a read/write version of pfSense (not the embedded for ALIX or Soekris since filesystem is read only).

Here is the bug discussion on the pfSense forum: http://redmine.pfsense.org/issues/1938

[pstutz]

Thank you for your guidance. I have confirmed that it's working ask desired now. I have rebuild my Splunk box and am running 4.2.4, build 110225.

[lguinn2 · Accepted solution]

I grabbed your log sample and saved it to a file on my system.
I created the following entries in my .conf files

#inputs.conf
[monitor:///opt/testdata/pfsense.log]
sourcetype=pfsense_pf

## props.conf
[pfsense_pf]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = match

I restarted Splunk and it found the file. It separated the data into 3 events. Timestamps were properly interpreted.
Of course, there are 6 lines of information, so each event consisted of two lines. Each event started with a line containing "match" as specified. It worked perfectly, as best I can tell.

I am running version 4.2.1 of Splunk. Next questions for you:

What results did you get?

What version of Splunk are you running?

Can you display the inputs.conf as well as the props.conf?