Getting Data In
Highlighted

How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

New Member

I have standard UDP logs from PFsense being sent to my Splunk server. However, I can't seem to get the Squid logs to Splunk. Does anyone have some quick advice on how to get these from a PFsense device?

Thanks

Apr 26 21:00:51 192.168.1.1 Apr 26 21:00:51 (squid-1): 1430110851.655 69 192.168.2.51 TCPMISS/200 638 GET http://geo.yahoo.com/p? - ORIGINALDST/208.71.44.31 image/gif
host = 192.168.1.1 source = udp:514 sourcetype = squid

Tags (3)
0 Karma
Highlighted

Re: How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

Splunk Employee
Splunk Employee

I've edited your question so that others can be helped here... Initially it seemed as if you weren't getting the data in, but in the end you're describing a problem with extracting the fields.

The fields are there but the app you're using isn't accounting for the syslog header. (the Squid App for Splunk Enterprise seems to have been updated in 2011 and not again...)

Also it's also counting on certain values being present in order to pick up the fields.. Look at the transforms.conf in the app and you'll see that the regex is not going to match your events.
So if you like, you can change the props.conf from:
REPORT-squid = squid
to

EXTRACT-squid =  \(squid-1\):\s+(?P<epoch_time>\d+\.\d+)\s(?P<duration>\d+)\s(?P<clientip>[^\s]+)\s(?P<action>[^\/]+)\/(?P<http_status>\d{3})\s(?P<bytes>\d+)\s+(?P<method>[^\s]+)\s(?P<uri>[^\s]+)\s(?P<unknown_value>[^\s]+)\s(?P<hierarchy_code>[^\/]+)\/(?<server_ip>\d+\.\d+\.\d+\.\d+)\s(?P<content_type>.+)

And you'll be able to get the fields. Or if you feel comfortable, you can alter the transforms, conf and use the regex I've given you (removing or accommodating for the FORMAT directive.

Note the savedsearches.conf in the app... all searches hang on the presence of the action field. You probably didn't have any fields.
action=* means... pull only events where the action field exists... that would be 0 until you extract the field.

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
Highlighted

Re: How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

New Member

Thanks for the info. I do have have the logs getting to Splunk but the problem seems that Squid for Splunk does not seem to be parsing the data correct for the dashboards.

Search: sourcetype=squid returns the following entries..

Apr 26 21:00:51 192.168.1.1 Apr 26 21:00:51 (squid-1): 1430110851.655 69 192.168.2.51 TCPMISS/200 638 GET http://geo.yahoo.com/p? - ORIGINALDST/208.71.44.31 image/gif
host = 192.168.1.1 source = udp:514 sourcetype = squid

0 Karma
Highlighted

Re: How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

Splunk Employee
Splunk Employee

You need to check the source of the file. Seems you are logging the Date and timestamp twice. Typically this is a r/syslog/-ng related issue for the input.

You can check the props for the squid source and change the parsing rules so it ignores the second date and timestamp, or fix the source...

0 Karma
Highlighted

Re: How to to extract fields from Squid logs to Splunk from PFsense Firewall using 2.2.1?

Splunk Employee
Splunk Employee

It's worse than that Eric... the app is very old and is written were it MUSt have all the fields in order for the transforms to work ( all in a row). On top of the syslog header... the event is also missing three or four fields (and I can't tell what the missing one is because there should be nothing there according to the transforms... ) it also doesn't account for the existence of the timestamp at all and begins with a ^ and expects the duration to be there... even without syslog, that wouldn't work, because the timestamp is going to be there either way... 😕

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!