Splunk Add-on for Blue Coat ProxySG: Has anyone have props.and transforms to work properly for Bluecoat 6.7.3.5 log formatting?

[mshakeb]

Hi Experts
Splunk Add-on for Blue Coat ProxySG: Has anyone gotten the props and transforms to work properly for Bluecoat 6.7.3.5 formatting, I have applied the 6.6.x.x on props and transforms but could not see the field extraction Properly. Many fields are missing. please advice

[kiyohito]

I had the same problem for bluecoat:proxysg:access:syslog. It's because Splunk Add-on for Blue Coat ProxySG Version 3.5.0 does not catch up with SGOS 6.7.

I'm not sure but found two issues for Add-on:

1. Regular expression error for cs-categories: "Technology/Internet;Web Ads/Analytics" was splited into "Tech....Web" and "Ads/Analytics".
2. Missing x-bluecoat-application-groups field

Solution is following. So far it works for me(Splunk 7.2.6, SGOS 6.7.2.1).

Add to transforms.conf

[auto_kv_for_bluecoat_v6_7_x]

REGEX = ^(?:"([^"]+)"|([^"]\S*))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"|(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*$

FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 s_supplier_name::$13 s_supplier_name::$14 s_supplier_ip::$15 s_supplier_ip::$16 s_supplier_country::$17 s_supplier_country::$18 s_supplier_failures::$19 s_supplier_failures::$20 x_exception_id::$21 x_exception_id::$22 sc_filter_result::$23 sc_filter_result::$24 cs_categories::$25 cs_categories::$26 cs_Referer::$27 cs_Referer::$28 sc_status::$29 sc_status::$30 s_action::$31 s_action::$32 cs_method::$33 cs_method::$34 rs_Content_Type::$35 rs_Content_Type::$36 cs_uri_scheme::$37 cs_uri_scheme::$38 cs_host::$39 cs_host::$40 cs_uri_port::$41 cs_uri_port::$42 cs_uri_path::$43 cs_uri_path::$44 cs_uri_query::$45 cs_uri_query::$46 cs_uri_extension::$47 cs_uri_extension::$48 cs_User_Agent::$49 cs_User_Agent::$50 s_ip::$51 s_ip::$52 sc_bytes::$53 sc_bytes::$54 cs_bytes::$55 cs_bytes::$56 x_virus_id::$57 x_virus_id::$58 x_bluecoat_application_name::$59 x_bluecoat_application_name::$60 x_bluecoat_application_operation::$61 x_bluecoat_application_operation::$62 x-bluecoat-application-groups::$63 x-bluecoat-application-groups::$64 cs_threat_risk::$65 cs_threat_risk::$66 x_bluecoat_transaction_uuid::$67 x_bluecoat_transaction_uuid::$68 x_icap_reqmod_header::$69 x_icap_reqmod_header::$70 x_icap_respmod_header::$71 x_icap_respmod_header::$72

Add to props.conf

# Supports Bluecoat 6.7 field format
REPORT-auto_kv_for_bluecoat_v6_7_x = auto_kv_for_bluecoat_v6_7_x

I hope it help someone and Add-on be update.

[joshuaonsecurit]

Is it possible to share your Bluecoat log format? I'm struggling to get the extract working.

[bsanta201]

Hello, Does this regex work with IVP6? if not does anyone have an updated regex that will work? I am not having any luck with IVP regex

[bsanta201]

Here is what I have (?:"([^"]+)"|([^"]\S*))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"|(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}))\s+(?:([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))|(?:[0-9]{1,3}.){3}[0-9]{1,3}\b))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*

[kiyohito]

Hi,

Log: main
Log format: bcreportermain_v1

date time time-taken c-ip cs-username cs-auth-group s-supplier-name s-supplier-ip s-supplier-country s-supplier-failures x-exception-id sc-filter-result cs-categories cs(Referer)  sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bluecoat-application-name x-bluecoat-application-operation x-bluecoat-application-groups cs-threat-risk x-bluecoat-transaction-uuid x-icap-reqmod-header(X-ICAP-Metadata) x-icap-respmod-header(X-ICAP-Metadata)

Ref: onfigure logging in your Blue Coat ProxySG appliance for the Splunk Add-on for Symantec Blue Co...

[sdchakraborty]

Hi,

Can you provide a sample log.

Sid

[mshakeb]

please find the logs :

Apr 3 03:14:48 133.22.00.00 2019-04-03 07:14:48 74 10.130.122.151 Jastaniahfm aljfs\Group%20Internet%20Basic www.google.com 172.217.19.132 None - - OBSERVED "CRM-Access;Search Engines/Portals" https://www.google.com/ 204 TCP_NC_MISS POST text/html https www.google.com 443 /gen_204 ?atyp=csi&ei=-VekXNPyG66PlwT_hZDYDg&s=newtab&t=all&action=update&conn=onchange&ima=1&ime=1&imeb=0&imeo=0&wh=618&scp=0&net=dl.1650,ect.4g,rtt.150&mem=ujhs.10,tjhs.11,jhsl.2330,dm.8&sto=&sys=hc.4&rt=xhr.342,aft.4,cst.0,dnst.0,rqst.70,rspt.21,rqstt.21,unt.21,cstt.21,dit.159&zx=1554275688237 - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" 133.22.00.001 306 1429 - "none" "none" "none" 1 7ee0b471858c027d-00000000228366b2-000000005ca45d68 - -

Apr 3 03:14:48 103.33.33.3 2019-04-03 07:14:48 68 10.0.51.51 Karimykd aljfs\Group%20Internet%20No%20Restriction ctldl.windowsupdate.com 10.111.000.241 None - - OBSERVED "White list website;Non-Viewable/Infrastructure" - 304 TCP_MISS GET application/vnd.ms-cab-compressed http ctldl.windowsupdate.com 80 /msdownload/update/v3/static/trustedr/en/disallowedcertstl.cab ?5fae3ccb4e56dcf8 cab "Microsoft-CryptoAPI/6.1" 10.22.22.22 386 322 - "Microsoft Update" "Update Software" "none" 1 7ee0b471858c027d-00000000228366af-000000005ca45d68 - "{ %22expect_sandbox%22: false }"