All Apps and Add-ons

Bluecoat ProxySG logs parsing properly

Loves-to-Learn Lots

Hi experts

Bluecoat proxysg logs are not parsing properly, we are sending  logs from Bluecoat proxy to syslog-ng server in W3C format. we notice that the bluecoat proxy logs itself does not sending the log headers. Does TA Require header in logs. please need help

 

Thanking in advance

Labels (1)
0 Karma

Champion

@mshakeb ,

There is no standard TA for bluecoat proxy which works for all versions of bluecoat proxy.

log format is changed based on version and how you format the log in the proxy.

if you can share sample log or the variables used to format the log in bluecoat (preferred) is helpful to help you.

————————————
If this helps, give a like below.
0 Karma

Loves-to-Learn Lots

Hi,

we are forwarding W3C ELEF format from Bluecoat proxysg to syslog-ng. we have notice on syslog-ng server  bluecoat logs are not having log headers. please see the log sample

 

Aug 9 12:00:00 10.0.xx.xx 2020-08-09 09:00:00 1 10.xx.0.10 aaxxxx yyyDomain\xcyv%20XXXXXXX%20Internet%2Group 10.0.xx.xxx 10.0.xx.xxx None - - OBSERVED "Youtube;Audio/Video Clips" https://www.youtube.com/ 200 TCP_HIT GET text/plain https r6---sn-4wgd.googlevideo.com 443 /videoplayback ?expire=15969881&ei=nbEvX9yLA4OWWMuDg5AO&ip=98.220.xx.xxx&id=oAJ4JFOEDAszlxnibA285uN6_lOZUonUhoyA44Rb2mE6&itag=396&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%cccv%2C395%2C396%2C397%2C398%2C399&source=youtube95390&req_id=53b73fd85edb82c5&altitags=395%2C394&rn=541668&rbuf=0 - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/10.0.0.xxx Safari/537.36" 10.x.x.000 2113 1530 - "YouTube" "Play Video" "Video Hosting" 2 a954beb9a-00000000b2ac6060-000000005f2fbb10 - -

0 Karma

Champion

download TA from https://splunkbase.splunk.com/app/2758/ 

use below props.conf and transforms.conf in local directory. make sure your sourctype of bluecoat logs is matching with "bluecoat:proxysg:access:syslog"

props.conf

 

 

[bluecoat:proxysg:access:syslog]
pulldown_type = true
category = Network & Security
description = Data from Blue Coat ProxySG in W3C ELFF format thru syslog
KV_MODE = none
SHOULD_LINEMERGE = false
EVENT_BREAKER_ENABLE=true
MAX_DAYS_AGO = 10951

REPORT-categories = bluecoat_categories
REPORT-bluecoat_custom = REPORT-bluecoat_custom

FIELDALIAS-cookie           = cs_Cookie as cookie
FIELDALIAS-duration         = time_taken as duration
FIELDALIAS-src=c_ip as src
FIELDALIAS-src_port         = c_port as src_port
FIELDALIAS-user             = cs_username as user
FIELDALIAS-http_referrer    = cs_Referer as http_referrer
FIELDALIAS-status           = sc_status as status
FIELDALIAS-action           = s_action as vendor_action
FIELDALIAS-http_method      = cs_method as http_method
FIELDALIAS-content_type     = rs_Content_Type as http_content_type
FIELDALIAS-dest_host        = cs_host as dest_host
FIELDALIAS-dest_port        = s_port as dest_port
FIELDALIAS-user_agent       = cs_User_Agent as http_user_agent
FIELDALIAS-dest_ip          = cs_ip as dest_ip
FIELDALIAS-dvc              = s_ip as dvc
FIELDALIAS-bytes_in         = sc_bytes as bytes_in
FIELDALIAS-bytes_out        = cs_bytes as bytes_out
FIELDALIAS-uri_path         = cs_uri_path as uri_path
FIELDALIAS-uri_query        = cs_uri_query as uri_query
FIELDALIAS-protocol         = cs_protocol as protocol
FIELDALIAS-packets_in       = c_pkts_received as packets_in
FIELDALIAS-session_id       = s_session_id as session_id

EVAL-app = "Blue Coat ProxySG"
EVAL-dest = coalesce(dest_ip, dest_host)
EVAL-bytes = bytes_in + bytes_out
EVAL-url = coalesce(cs_uri, if(isnull(cs_uri_scheme) OR (cs_uri_scheme=="-"), "", cs_uri_scheme+"://") + cs_host + cs_uri_path + if(isnull(cs_uri_query) OR (cs_uri_query == "-"), "", cs_uri_query))
EVAL-product = "ProxySG"
EVAL-vendor = "Blue Coat"
EVAL-vendor_product = "Blue Coat ProxySG"
# Eval action to blocked if sc_filter_result is DENIED
EVAL-action = case(sc_filter_result=="DENIED","blocked")
LOOKUP-vendor_traffic_action = bluecoat_proxy_action_lookup vendor_action OUTPUTNEW action, transport

 

 

transforms.conf

Note: the fields after field4 may not match with your field format, identify and replace right field names in below FIELDS

 

 

[REPORT-bluecoat_custom]
DELIMS = " "
#Below was used before remove default string appended by syslog-ng
FIELDS = "field1","field2","field3","field4","date","time","time-taken","c-ip","cs-username","cs-auth-group","s-supplier-name","s-supplier-ip","s-supplier-country","s-supplier-failures","x-exception-id","sc-filter-result","cs-categories","cs(Referer)","sc-status","s-action","cs-method","rs(Content-Type)","cs-uri-scheme","cs-host","cs-uri-port","cs-uri-path","cs-uri-query","cs-uri-extension","cs(User-Agent)","s-ip","sc-bytes","cs-bytes","x-virus-id","x-bluecoat-application-name","x-bluecoat-application-operation","x-bluecoat-application-groups","cs-threat-risk","x-bluecoat-transaction-uuid","x-icap-reqmod-header(X-ICAP-Metadata)","x-icap-respmod-header(X-ICAP-Metadata)"

[bluecoat_proxy_action_lookup]
filename = bluecoat_proxy_actions.csv
case_sensitive_match = false

[bluecoat_categories]
SOURCE_KEY = cs_categories
REGEX = (?<category>[^;]+)
MV_ADD = true

 

 

The reason why default transforms doesn't work is because you are collecting events through syslog server, and syslog server is appending data , time, and host to actual message coming from proxy.

up vote if this works.

————————————
If this helps, give a like below.
0 Karma

Champion

Hi

General rules is "Don't remove (or change) configurations on default folder! Just add needed configuration files to local folder and only needed parts which you must change"

r. Ismo

0 Karma

Loves-to-Learn Lots

@thambisetty 

Bluecoat proxy version 6.7

 

format : "date","time","time-taken","c-ip","cs-username","cs-auth-group","s-supplier-name","s-supplier-ip","s-supplier-country","s-supplier-failures","x-exception-id","sc-filter-result","cs-categories","cs(Referer)","sc-status","s-action","cs-method","rs(Content-Type)","cs-uri-scheme","cs-host","cs-uri-port","cs-uri-path","cs-uri-query","cs-uri-extension","cs(User-Agent)","s-ip","sc-bytes","cs-bytes","x-virus-id","x-bluecoat-application-name","x-bluecoat-application-operation","x-bluecoat-application-groups","cs-threat-risk","x-bluecoat-transaction-uuid","x-icap-reqmod-header(X-ICAP-Metadata)","x-icap-respmod-header(X-ICAP-Metadata)"

0 Karma

Champion

@soutamo , updated my answer.

@mshakeb ,

please use updated answer, that should work for you now.

up vote, if that works.

————————————
If this helps, give a like below.
0 Karma

Champion

Can you search “fields” in your proxy logs where you will see header of the logs, bluecoat version etc. if you found that, please share here.

————————————
If this helps, give a like below.
0 Karma