Getting Data In

Splunk Add-on for Blue Coat ProxySG: Has anyone have props.and transforms to work properly for Bluecoat 6.7.3.5 log formatting?

mshakeb
Loves-to-Learn Lots

Hi Experts
Splunk Add-on for Blue Coat ProxySG: Has anyone gotten the props and transforms to work properly for Bluecoat 6.7.3.5 formatting, I have applied the 6.6.x.x on props and transforms but could not see the field extraction Properly. Many fields are missing. please advice

Tags (1)
0 Karma

kiyohito
Engager

I had the same problem for bluecoat:proxysg:access:syslog. It's because Splunk Add-on for Blue Coat ProxySG Version 3.5.0 does not catch up with SGOS 6.7.

I'm not sure but found two issues for Add-on:

  1. Regular expression error for cs-categories: "Technology/Internet;Web Ads/Analytics" was splited into "Tech....Web" and "Ads/Analytics".
  2. Missing x-bluecoat-application-groups field

Solution is following. So far it works for me(Splunk 7.2.6, SGOS 6.7.2.1).

Add to transforms.conf


[auto_kv_for_bluecoat_v6_7_x]
REGEX = ^(?:"([^"]+)"|([^"]\S*))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d+)"|(\d+))\s+(?:"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"|(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*$
FORMAT = date::$1 date::$2 time::$3 time::$4 time_taken::$5 time_taken::$6 c_ip::$7 c_ip::$8 cs_username::$9 cs_username::$10 cs_auth_group::$11 cs_auth_group::$12 s_supplier_name::$13 s_supplier_name::$14 s_supplier_ip::$15 s_supplier_ip::$16 s_supplier_country::$17 s_supplier_country::$18 s_supplier_failures::$19 s_supplier_failures::$20 x_exception_id::$21 x_exception_id::$22 sc_filter_result::$23 sc_filter_result::$24 cs_categories::$25 cs_categories::$26 cs_Referer::$27 cs_Referer::$28 sc_status::$29 sc_status::$30 s_action::$31 s_action::$32 cs_method::$33 cs_method::$34 rs_Content_Type::$35 rs_Content_Type::$36 cs_uri_scheme::$37 cs_uri_scheme::$38 cs_host::$39 cs_host::$40 cs_uri_port::$41 cs_uri_port::$42 cs_uri_path::$43 cs_uri_path::$44 cs_uri_query::$45 cs_uri_query::$46 cs_uri_extension::$47 cs_uri_extension::$48 cs_User_Agent::$49 cs_User_Agent::$50 s_ip::$51 s_ip::$52 sc_bytes::$53 sc_bytes::$54 cs_bytes::$55 cs_bytes::$56 x_virus_id::$57 x_virus_id::$58 x_bluecoat_application_name::$59 x_bluecoat_application_name::$60 x_bluecoat_application_operation::$61 x_bluecoat_application_operation::$62 x-bluecoat-application-groups::$63 x-bluecoat-application-groups::$64 cs_threat_risk::$65 cs_threat_risk::$66 x_bluecoat_transaction_uuid::$67 x_bluecoat_transaction_uuid::$68 x_icap_reqmod_header::$69 x_icap_reqmod_header::$70 x_icap_respmod_header::$71 x_icap_respmod_header::$72

Add to props.conf

# Supports Bluecoat 6.7 field format
REPORT-auto_kv_for_bluecoat_v6_7_x = auto_kv_for_bluecoat_v6_7_x

I hope it help someone and Add-on be update.

joshuaonsecurit
New Member

Is it possible to share your Bluecoat log format? I'm struggling to get the extract working.

0 Karma

bsanta201
New Member

Hello, Does this regex work with IVP6? if not does anyone have an updated regex that will work? I am not having any luck with IVP regex

0 Karma

bsanta201
New Member

Here is what I have (?:"([^"]+)"|([^"]\S*))\s+(?:"(\d{1,2}:\d{1,2}:\d{1,2})"|(\d{1,2}:\d{1,2}:\d{1,2}))\s+(?:"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})"|(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}))\s+(?:([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]).){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))|(?:[0-9]{1,3}.){3}[0-9]{1,3}\b))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s+(?:"([^"]+)"|([^"]\S*))\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*(?:"([^"]+)"|([^"]\S*))?\s*

0 Karma

kiyohito
Engager

Hi,

Log: main
Log format: bcreportermain_v1


date time time-taken c-ip cs-username cs-auth-group s-supplier-name s-supplier-ip s-supplier-country s-supplier-failures x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bluecoat-application-name x-bluecoat-application-operation x-bluecoat-application-groups cs-threat-risk x-bluecoat-transaction-uuid x-icap-reqmod-header(X-ICAP-Metadata) x-icap-respmod-header(X-ICAP-Metadata)

Ref: onfigure logging in your Blue Coat ProxySG appliance for the Splunk Add-on for Symantec Blue Co...

0 Karma

sdchakraborty
Contributor

Hi,

Can you provide a sample log.

Sid

0 Karma

mshakeb
Loves-to-Learn Lots

please find the logs :

Apr 3 03:14:48 133.22.00.00 2019-04-03 07:14:48 74 10.130.122.151 Jastaniahfm aljfs\Group%20Internet%20Basic www.google.com 172.217.19.132 None - - OBSERVED "CRM-Access;Search Engines/Portals" https://www.google.com/ 204 TCP_NC_MISS POST text/html https www.google.com 443 /gen_204 ?atyp=csi&ei=-VekXNPyG66PlwT_hZDYDg&s=newtab&t=all&action=update&conn=onchange&ima=1&ime=1&imeb=0&imeo=0&wh=618&scp=0&net=dl.1650,ect.4g,rtt.150&mem=ujhs.10,tjhs.11,jhsl.2330,dm.8&sto=&sys=hc.4&rt=xhr.342,aft.4,cst.0,dnst.0,rqst.70,rspt.21,rqstt.21,unt.21,cstt.21,dit.159&zx=1554275688237 - "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" 133.22.00.001 306 1429 - "none" "none" "none" 1 7ee0b471858c027d-00000000228366b2-000000005ca45d68 - -

Apr 3 03:14:48 103.33.33.3 2019-04-03 07:14:48 68 10.0.51.51 Karimykd aljfs\Group%20Internet%20No%20Restriction ctldl.windowsupdate.com 10.111.000.241 None - - OBSERVED "White list website;Non-Viewable/Infrastructure" - 304 TCP_MISS GET application/vnd.ms-cab-compressed http ctldl.windowsupdate.com 80 /msdownload/update/v3/static/trustedr/en/disallowedcertstl.cab ?5fae3ccb4e56dcf8 cab "Microsoft-CryptoAPI/6.1" 10.22.22.22 386 322 - "Microsoft Update" "Update Software" "none" 1 7ee0b471858c027d-00000000228366af-000000005ca45d68 - "{ %22expect_sandbox%22: false }"

0 Karma
Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...