topic Re: Bluecoat ProxySG logs parsing properly in All Apps and Add-ons

Bluecoat ProxySG logs parsing properly

mshakeb — Mon, 10 Aug 2020 14:15:48 GMT

Hi experts

Bluecoat proxysg logs are not parsing properly, we are sending logs from Bluecoat proxy to syslog-ng server in W3C format. we notice that the bluecoat proxy logs itself does not sending the log headers. Does TA Require header in logs. please need help

Thanking in advance

Re: Bluecoat ProxySG logs parsing properly

thambisetty — Mon, 10 Aug 2020 14:23:05 GMT

@mshakeb ,

There is no standard TA for bluecoat proxy which works for all versions of bluecoat proxy.

log format is changed based on version and how you format the log in the proxy.

if you can share sample log or the variables used to format the log in bluecoat (preferred) is helpful to help you.

Re: Bluecoat ProxySG logs parsing properly

mshakeb — Mon, 10 Aug 2020 14:45:35 GMT

Hi,

we are forwarding W3C ELEF format from Bluecoat proxysg to syslog-ng. we have notice on syslog-ng server bluecoat logs are not having log headers. please see the log sample

Aug 9 12:00:00 10.0.xx.xx 2020-08-09 09:00:00 1 10.xx.0.10 aaxxxx yyyDomain\xcyv%20XXXXXXX%20Internet%2Group 10.0.xx.xxx 10.0.xx.xxx None - - OBSERVED "Youtube;Audio/Video Clips" https://www.youtube.com/ 200 TCP_HIT GET text/plain https r6---sn-4wgd.googlevideo.com 443 /videoplayback ?expire=15969881&ei=nbEvX9yLA4OWWMuDg5AO&ip=98.220.xx.xxx&id=oAJ4JFOEDAszlxnibA285uN6_lOZUonUhoyA44Rb2mE6&itag=396&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%cccv%2C395%2C396%2C397%2C398%2C399&source=youtube95390&req_id=53b73fd85edb82c5&altitags=395%2C394&rn=541668&rbuf=0 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/10.0.0.xxx Safari/537.36" 10.x.x.000 2113 1530 - "YouTube" "Play Video" "Video Hosting" 2 a954beb9a-00000000b2ac6060-000000005f2fbb10 - -

Re: Bluecoat ProxySG logs parsing properly

thambisetty — Mon, 10 Aug 2020 14:52:09 GMT

Can you search “fields” in your proxy logs where you will see header of the logs, bluecoat version etc. if you found that, please share here.

Re: Bluecoat ProxySG logs parsing properly

thambisetty — Tue, 11 Aug 2020 09:45:43 GMT

download TA from https://splunkbase.splunk.com/app/2758/

use below props.conf and transforms.conf in local directory. make sure your sourctype of bluecoat logs is matching with "bluecoat:proxysg:access:syslog"

props.conf

[bluecoat:proxysg:access:syslog] pulldown_type = true category = Network & Security description = Data from Blue Coat ProxySG in W3C ELFF format thru syslog KV_MODE = none SHOULD_LINEMERGE = false EVENT_BREAKER_ENABLE=true MAX_DAYS_AGO = 10951 REPORT-categories = bluecoat_categories REPORT-bluecoat_custom = REPORT-bluecoat_custom FIELDALIAS-cookie = cs_Cookie as cookie FIELDALIAS-duration = time_taken as duration FIELDALIAS-src=c_ip as src FIELDALIAS-src_port = c_port as src_port FIELDALIAS-user = cs_username as user FIELDALIAS-http_referrer = cs_Referer as http_referrer FIELDALIAS-status = sc_status as status FIELDALIAS-action = s_action as vendor_action FIELDALIAS-http_method = cs_method as http_method FIELDALIAS-content_type = rs_Content_Type as http_content_type FIELDALIAS-dest_host = cs_host as dest_host FIELDALIAS-dest_port = s_port as dest_port FIELDALIAS-user_agent = cs_User_Agent as http_user_agent FIELDALIAS-dest_ip = cs_ip as dest_ip FIELDALIAS-dvc = s_ip as dvc FIELDALIAS-bytes_in = sc_bytes as bytes_in FIELDALIAS-bytes_out = cs_bytes as bytes_out FIELDALIAS-uri_path = cs_uri_path as uri_path FIELDALIAS-uri_query = cs_uri_query as uri_query FIELDALIAS-protocol = cs_protocol as protocol FIELDALIAS-packets_in = c_pkts_received as packets_in FIELDALIAS-session_id = s_session_id as session_id EVAL-app = "Blue Coat ProxySG" EVAL-dest = coalesce(dest_ip, dest_host) EVAL-bytes = bytes_in + bytes_out EVAL-url = coalesce(cs_uri, if(isnull(cs_uri_scheme) OR (cs_uri_scheme=="-"), "", cs_uri_scheme+"://") + cs_host + cs_uri_path + if(isnull(cs_uri_query) OR (cs_uri_query == "-"), "", cs_uri_query)) EVAL-product = "ProxySG" EVAL-vendor = "Blue Coat" EVAL-vendor_product = "Blue Coat ProxySG" # Eval action to blocked if sc_filter_result is DENIED EVAL-action = case(sc_filter_result=="DENIED","blocked") LOOKUP-vendor_traffic_action = bluecoat_proxy_action_lookup vendor_action OUTPUTNEW action, transport

transforms.conf

Note: the fields after field4 may not match with your field format, identify and replace right field names in below FIELDS

[REPORT-bluecoat_custom] DELIMS = " " #Below was used before remove default string appended by syslog-ng FIELDS = "field1","field2","field3","field4","date","time","time-taken","c-ip","cs-username","cs-auth-group","s-supplier-name","s-supplier-ip","s-supplier-country","s-supplier-failures","x-exception-id","sc-filter-result","cs-categories","cs(Referer)","sc-status","s-action","cs-method","rs(Content-Type)","cs-uri-scheme","cs-host","cs-uri-port","cs-uri-path","cs-uri-query","cs-uri-extension","cs(User-Agent)","s-ip","sc-bytes","cs-bytes","x-virus-id","x-bluecoat-application-name","x-bluecoat-application-operation","x-bluecoat-application-groups","cs-threat-risk","x-bluecoat-transaction-uuid","x-icap-reqmod-header(X-ICAP-Metadata)","x-icap-respmod-header(X-ICAP-Metadata)" [bluecoat_proxy_action_lookup] filename = bluecoat_proxy_actions.csv case_sensitive_match = false [bluecoat_categories] SOURCE_KEY = cs_categories REGEX = (?<category>[^;]+) MV_ADD = true

The reason why default transforms doesn't work is because you are collecting events through syslog server, and syslog server is appending data , time, and host to actual message coming from proxy.

up vote if this works.

Re: Bluecoat ProxySG logs parsing properly

isoutamo — Mon, 10 Aug 2020 21:04:50 GMT

General rules is "Don't remove (or change) configurations on default folder! Just add needed configuration files to local folder and only needed parts which you must change"

r. Ismo

Re: Bluecoat ProxySG logs parsing properly

mshakeb — Tue, 11 Aug 2020 09:31:55 GMT

@thambisetty

Bluecoat proxy version 6.7

format : "date","time","time-taken","c-ip","cs-username","cs-auth-group","s-supplier-name","s-supplier-ip","s-supplier-country","s-supplier-failures","x-exception-id","sc-filter-result","cs-categories","cs(Referer)","sc-status","s-action","cs-method","rs(Content-Type)","cs-uri-scheme","cs-host","cs-uri-port","cs-uri-path","cs-uri-query","cs-uri-extension","cs(User-Agent)","s-ip","sc-bytes","cs-bytes","x-virus-id","x-bluecoat-application-name","x-bluecoat-application-operation","x-bluecoat-application-groups","cs-threat-risk","x-bluecoat-transaction-uuid","x-icap-reqmod-header(X-ICAP-Metadata)","x-icap-respmod-header(X-ICAP-Metadata)"

Re: Bluecoat ProxySG logs parsing properly

thambisetty — Tue, 11 Aug 2020 09:46:39 GMT

@isoutamo , updated my answer.

@mshakeb ,

please use updated answer, that should work for you now.

up vote, if that works.