How to extract fields from bluecoat logs?

[TheGU]

How to extract the log example below:

2010-09-29 16:23:44 2 172.16.106.54 exam.ple Filter-ID==4 - OBSERVED "Search Engines/Portals;News/Media" http://market.example.com/product/1439306 200

I got this from bluecoat server. I can't use DELIMS=" " to extract KV because some fields have a " " (spacebar) too.

for extraction example :

• 2010-09-29
• 16:23:44
• 2
• 172.16.106.54
• exam.ple
• Filter-ID==4
• -
• OBSERVED
• "Search Engines/Portals;News/Media" <<<<<<<<<< this field make a problem!!!
• http://market.example.com/product/1439306
• 200

[Peterman]

---

@TheGU wrote:

How to extract the log example below:

2010-09-29 16:23:44 2 172.16.106.54 exam.ple Filter-ID==4 - OBSERVED "Search Engines/Portals;News/Media" http://market.example.com/product/1439306 200

I got this from bluecoat server. I can't use DELIMS=" " to extract KV because some fields have a " " (spacebar) too.

for extraction example :

• 2010-09-29
• 16:23:44
• 2
• 172.16.106.54
• exam.ple
• Filter-ID==4
• -
• OBSERVED
• "Search Engines/Portals;News/Media" <<<<<<<<<< this field make a problem!!!
• http://market.example.com/product/1439306
• 200

---

bluecoat server is awsome man 

[ziegfried · Accepted solution]

You could use a regular expression to extract the fields. Something like this:

props.conf

[your_sourcetype]
EXTRACT-fields=^[\d\-]+) [\d:]+ (?<FIELD_1>\d+) (?<clientip>[\d\.]+) (?<domain>\S+) Filter\-ID==(?<filter_id>\S+) "(?<url_category>[^"]+)" (?<url>\S+) (?<status>\d+)

[TheGU]

I found that transaction may be up to 2kB. Very hard to write a regex to cover all of them. However your answer is Ok. Thanks.